by Head of Search
Andrew Girdwood
Google has opened the doors on a large spidering experiment. In the search engine's own words the sitemap beta, "will either fail miserably, or succeed beyond our wildest dreams".Andrew Girdwood
The premise is simple enough; a hosted XML file which lists all the pages from your site and asks Googlebot to visit them. The technique favours sites where URLs come and go quickly as the sitemap can be updated and Google told of the change. The sitemap is a great asset to overcoming spider accessibility issues too.
The catch is that the sitemap does nothing to help promote pages within Google. Webmasters can set a priority level for each page but this action "suggests" an importance for inclusion (would it be terrible if this page wasn't in Google?) and does not set an order of importance for search results.
There are practical limitations too. There is a maximum size any sitemap can be and although this can be overcome by having multiple sitemaps there is a maximum number of sitemaps which any one site can have.
The new system has been released under the Creative Commons license and this will make it easier for other search engines to tap into the technology and make use of the sitemap XML files too.
A free XML feed for search inclusion could be bad news for Yahoo!. The rival search engine currently charges a per-URL inclusion fee on top of a cost-per-click for SiteMatch and a cost-per-click for SiteMatch Xchange.
The initial signs already show great enthusiasm for the Sitemap program and monikers like "RSS for the entire site" are rising to the fore.
