The Robot Exclusion Standard also known as the Robots Exclusion Protocol or robots.txt protocol. By default search engine frequently fetching the content from your site and show in search result. To avoid your site or some pages from the search engine, you can use robots.txt file.
Why robots.txt fileThis robots.txt text file contains the instructions in a specific format. Robots that choose to follow the instructions try to fetch this file and read the instructions before fetching any other file from the web site. Simply visit or not the page.
Allow indexing of everything
User-agent: * Disallow:
User-agent: * Allow: /
Disallow indexing of everything
User-agent: * Disallow: /
Disawllow indexing of a specific folder
User-agent: * Disallow: /folder/ Disallow: /tempfolder/ Disallow: /employee-image/
Disawllow indexing of a specific file
User-agent: * Disallow: /filename.html Disallow: /tempfolder/tempfile.html
Disallow Googlebot or specific robot from indexing
User-agent: Googlebot Disallow: /