03 September 2009 | Author: M. Thomson SEO & Affiliate Consultant

Visit-Time Robots.txt Directive – use 503 instead

The Robots Exclusion Protocol (or robots.txt) is a tool used by search engines to control the ways in which search engine spiders crawl a website. Evolving alongside new technologies such as the Sitemap Protocol, the Robots Exclusion Protocol may see future additions to its list of supported contents.

Visit-Time

A visit-time directive would be a valuable inclusion within the Extended Standard for Robot Exclusion. The visit-time directive instructs search engine spiders on the ways in which they can crawl a website. The contents supplied include a start and end date followed by a UTC (Coordinated Universal Time).

Visit-Time: 2300-0500 UTC+1

Bigmouthmedia is always interested in new web prospects and believes that, while this directive would be useful for many users, it may prove inconvenient for search engines, which face enough obstacles already when crawling the web.

Users of the proposed visit-time directive shouldn't hold their breath awaiting support from Bing, Google or Yahoo! A bigmouthmedia test has demonstrated that none of the big three search engines are currently following this directive.

Controlling Visit Times

If your server currently experiences crawling problems during peak visit times, Google employee John Mueller (JohnMu) recommends that, instead of hoping that Googlebot will follow the visit-time robots.txt directive, users instead turn to the HTTP Error code 503 service unavailable directive.

Mr Mueller states: "One thing you can do is to encourage Googlebot (and other crawlers) to not visit your site at busy times by returning a 503 HTTP result code. This tells us that you currently can't serve the content, but that we should come back at some later time."

The 503 response will advise Googlebot and others to return and try to access the website at a later time, though users are not able to specify this time. Mr Mueller confirms: "we ignore any "Retry-After" information at the moment."

If Google begins testing the support for the visit-time directive in the future, the bigmouthmedia team won't be overly surprised.. Only recently, a test was carried out to monitor the support the unofficial NOINDEX robots.txt directive, leading to promising results, and Google claims it undertook over three hundred algorithm changes last year.