The Robot Exclusion Standard also known as the Robots Exclusion Protocol or robots.txt protocol. By default search engine frequently fetching the content from your site and show in search result. To avoid your site or some pages from the search engine, you can use robots.txt file.
Why robots.txt file
This robots.txt text file contains the instructions in a specific format. Robots that choose to follow the instructions try to fetch this file and read the instructions before fetching any other file from the web site. Simply visit or not the page.
Allow indexing of everything
Disallow indexing of everything
Disawllow indexing of a specific folder
Disawllow indexing of a specific file
Disallow Googlebot or specific robot from indexing