This file is located at the root of a Website, and it is named "robots.txt". It indicates directives to Robots, which are automated computer programs used by Search Engines to "crawl" the Internet and store information about each Webpage on the Internet. The directives in this file can tell a Robot to not index certain Webpages, which might be duplicate or useless Webpages to a Search Engine. Some common Robots are GoogleBot™ (Google), Slurp™ (Yahoo), BingBot™ (Microsoft), and SEOENGBot™ (SEOEngine.com). SEOENGBot™ follows the robot standard.
Robots.txt
Allow SEOENGBot™ to Crawl your Website!
SEOENGBot™ is a very responsible Spider-Bot. It follows all
robots.txt directives, and ensures that the Website it is crawling is not harmed by excessive bandwidth requests. To ensure that SEOENGBot™ can analyze your Website, place the following code into your robots.txt file:
# SEOENGBot knows where to go.
User-agent: seoengbot
Disallow:
Ensure that SEOENGBot™ can crawl your Website by adding this to robots.txt
