robots.txt

The Robot Exclusion Standard also known as the Robots Exclusion Protocol or robots.txt protocol.

By default search engine frequently fetching the content from your site and show in search result. To avoid your site or some pages from the search engine, you can use robots.txt file.

Why robots.txt file

This robots.txt text file contains the instructions in a specific format. Robots that choose to follow the instructions try to fetch this file and read the instructions before fetching any other file from the web site. Simply visit or not the page.

Allow indexing of everything

User-agent: *
Disallow:
User-agent: *
Allow: /

Disallow indexing of everything

User-agent: *
Disallow: /

Disawllow indexing of a specific folder

User-agent: *
Disallow: /folder/
Disallow: /tempfolder/
Disallow: /employee-image/

Disawllow indexing of a specific file

User-agent: *
Disallow: /filename.html
Disallow: /tempfolder/tempfile.html

Disallow Googlebot or specific robot from indexing

User-agent: Googlebot 
Disallow: /