"In addition to crawling the Internet faster, our new crawler is more efficient at visiting websites. As a result, site owners will notice as much as a 25% reduction in the number of requests and bandwidth consumed by the crawler.
"While transitioning to the new crawler over the past few weeks, we had been running both crawlers in tandem. In some cases, this increased the frequency of Yahoo! Search requests to websites. Now, with the new crawler in full production, we have turned off the old crawler and site owners should see a much lower crawl load without a loss in content coverage.
"With this change of behavior in the crawler, you may see some shuffling of the pages that are included in the index and some changes in ranking as well."
This update has been eagerly awaited and should aid Yahoo! in returning quality search results to users. However those who have concerns are invited to contact Yahoo! through a feedback form while any technical problems are to be handled by Yahoo! support team.
One feature of this update is the level of control users now have with regards to allowing the Slurp spider access various parts of the site including instructing it not to index specific content. The new update ensures that the slurp spider now takes robots.txt standards into account, essentially allowing users to turn off sections of their site to the spiders.
Robots.txt is a not a new technology, it has been a favored technique with search engine optimisation companies. By disallowing a page with robots.txt standards, the new Yahoo! Slurp spider will not read or use the contents of the page within its index. The search engine giant does, however, state that protected URLs may be included in the final SERPs as a 'thin' document without any text content. They also outline that any links or reference text from other public site which provide identifiable information regarding a URL may be indexed as part of their search engine coverage of a site.
As the update is now complete those who have seen their search engine ranks fluctuate should be confident that they will settle down. The ability for sites to Yo-Yo up and down Yahoo!'s index is common, especially during an update. It remains to be seen who has benefited from this update and who has lost rankings.
