Robots are your friend! Utilizing a Robots.txt File

Robots are your friend!

Creating a robots.txt file is simple, and gives you control over how your Website is crawled by Search Engines. You can use notepad or any text editor. Each entry consists of two lines:

User-Agent: [Spider or Bot name]
Disallow: [Directory or File Name]

The idea behind this concept is simple -- Webpages which you would like to keep private from the rest of the world (and Search Engines) should be handled via robots.txt. Spider-based Robots will ignore these Webpages and thus will not include any of Links or content on the Search Results.

Examples:

1. Excluding an individual file (page) from an individual Search Engine. You have a file (page), myprivatepage.htm, in a directory; private_pages, that you don’t want Google to index.
    
   User-Agent: Googlebot
   Disallow: /private_pages/myprivatepage.htm
    
2. If you want to exclude additional files (pages), simply add more Disallow lines.
   User-Agent: Googlebot
   Disallow: /private_pages/myprivatepage.htm
   Disallow: /private_pages/myprivatepage_2.htm
   Disallow: /private_pages/myprivatepage_3.htm
   Disallow: /private_pages/myprivatepage_4.htm
   
3. Excluding an entire section of your website from all Search Engines. You build a section of your website that you don’t want any search engine to index. This can also be helpful to use while part of your website is under construction, and you can remove it later when you’re ready to go live. The * means ALL search engines.
   
   User-Agent: *
   Disallow: /private_pages/
   
   The forward slash at the beginning and end of the directory name tells search engine you do not want any files (pages) in that directory indexed.
4. Allow all Search Engines to index your entire website. The * means ALL search engines. Simply leave the Disallow line empty, and the entire website will be open to indexing by all Search Engines.
   
   User-agent: *
   Disallow:
   
5. Allow no Search Engines to index any part of your website. The * means ALL search engines. Simply place a forward slash on the Disallow line.
   
   User-agent: *
   Disallow: /