robots.txt

The Robot Exclusion Standard also known as the Robots Exclusion Protocol or robots.txt protocol. By default search engine frequently fetching the content from your site and show in search result. To avoid your site or some pages from the search engine, you can use robots.txt file.

Why robots.txt file

This robots.txt text file contains the instructions in a specific format. Robots that choose to follow the instructions try to fetch this file and read the instructions before fetching any other file from the web site. Simply visit or not the page.

Allow indexing of everything


User-agent: *
Disallow:

User-agent: *
Allow: /

Disallow indexing of everything


User-agent: *
Disallow: /

Disawllow indexing of a specific folder


User-agent: *
Disallow: /folder/
Disallow: /tempfolder/
Disallow: /employee-image/

Disawllow indexing of a specific file


User-agent: *
Disallow: /filename.html
Disallow: /tempfolder/tempfile.html

Disallow Googlebot or specific robot from indexing


User-agent: Googlebot 
Disallow: /

 

Leave a Reply

Your email address will not be published. Required fields are marked *