by SEO & Affiliate Consultant
M. Thomson
Google Webmaster Tools is a free online service that allows webmasters to verify their website and view detailed reports about that website's visibility. Upon verification, a webmaster can advise Googlebot of a preferred crawl rate. This new crawl rate then affects the frequency of Googlebot's requests during the crawling process.M. Thomson
Recently, a handful of webmasters have been noticing that this setting is returning the following message:
"Your site has been assigned special crawl rate settings. You will not be able to change the crawl rate."
Webmasters with questions on this comment turned to the Google Webmaster Help Forum for answers.
A Google employee answered:
"The special crawl rate settings apply to sites where we are manually optimizing the crawl rate on our side. This is often due to feedback from the webmaster, the hoster or just generally because we've determined that the site can profit from a manual crawl rate setting by our engineers."
From this reply, it would indicate that Google engineers frequently override or manually apply a crawl rate to websites without the webmasters authorisation or knowledge. While bigmouthmedia believes Google's engineers have web sites' best interests in mind, but will this opaque approach to Google's internal system workings sway webmasters towards competing search engines like Yahoo! or newly launched Bing?
Other webmasters may interpret this comment in a similar fashion to Robin Hood's motto of "steal from the rich and give to the poor" - is Google's natural way of determining crawl rate being overridden by Google engineers on websites plentiful of crawl resource, and being given to websites with little?
The number of pages indexed by Google has increased rapidly over the last year - has the decision to hand edit crawl rates been a contributing factor? If so, it has certainly worked and despite the possibility of the search engine opening itself up to accusations of underhandedness, it seems that the strategy should have long-term benefits for webmasters.
