16 November 2006

Search Engine Sitemaps - Sitemaps now supported by top three search engines under a creative commons licence

Some positive news has been revealed in the SEO world today, namely that Google, MSN and Yahoo have all agreed to support a unified system of submitting web pages to their crawlers. Whilst Google launched their sitemap xml programme in June last year, (and Yahoo have taken similar steps with Site Explorer) this move marks the day when the other 'big two' have agreed that they will all now support the Google xml method via the new www.sitemaps.org destination.

• Google Sitemaps evolves into Search Engine Sitemaps!

Sitemaps were introduced and supported by Google on the 2nd of June, 2005. Initially released as a beta program, and without prior announcement or previews on Google Labs, Google's sitemaps programme was released under the 'Attribution / Share Alike' Creative Commons license with the prospect that other search engines may adopt the protocol.

As Google states, today:

"Last year we published the Sitemap 0.84 XML protocol as a free and easy way for webmasters to inform search engines about URLs on their web sites so that search engines can more effectively crawl them. We released it under the Attribution/Share Alike Creative Commons license in the hopes that other search engines would adopt the protocol too. And today, we're excited to announce that Yahoo! and Microsoft are joining us in officially supporting the Sitemap protocol."

• A recap on Google Sitemaps

The Google Sitemap - no, wait: the Search Engine Sitemap! - consists physically of a hosted XML file, generally named a sitemap.xml file. Think of this sitemap file as the antithesis of the robots.txt file. Robots.txt files are placed on the root folder of a domain to inform search engines of parts of the site which should not be spidered and included in search engines. A search engine sitemap should be similarly placed within a website and should indicated URLs that should be indexed. The sitemap.xml file is essentially a specially formulated set of instructions aimed specifically at search engine spiders with the aim of improving URL inclusion.

This does not mean that only those pages included in the XML file will be indexed, it just help draw the search engine spiders' attention to pages it would not find otherwise, using the traditional method of following links. The original Google sitemap protocol was released as the Sitemap 0.84 XML protocol.

• The new Search Engine Sitemap Protocol

"As part of this development, we're moving the protocol to a new namespace, www.sitemaps.org , and raising the version number to 0.9. The sponsoring companies will continue to collaborate on the protocol and publish enhancements on the jointly-maintained site sitemaps.org"

• The new Search Engine Sitemap Site: http://www.sitemaps.org /

• Are sitemaps always useful for search engine Spidering?

There has been some speculation on the internet that using a sitemap can actually harm your rankings; that a sitemap draws focus and attention to all parts of your site and to that effect a sitemap can harm your standing with Google. The reality is that there is no evidence to support this effect from sitemaps, and documented issues from sites on forums have often been found by the more experience SEO to contain a number of significant issues. The loading of a sitemap was often an easy action to attribute to a negative event after it - Post hoc ergo propter hoc!

• What do new search engine sitemap webmaster need to do?

"If you haven't submitted a Sitemap before, check out the documentation on www.sitemaps.org for information on creating one. You can submit your Sitemap file to Google using Google webmaster tools. See the documentation that Yahoo! and Microsoft provide for information about submitting to them."

• What about existing Google sitemap users?

Existing sitemaps users can rest easy. There is good news from Google for on-the-ball webmasters:

"If you've already submitted a Sitemap to Google using the previous namespace and version number, we'll continue to accept it."

This is excellent news for those of us regularly involved in attaining submission to each of the search engines, and who have to cater for different standards for each: as life just go a little easier. Of course, we also now have a new benchmark for search engine comparison: indexing wars!

Imagine the situation: a new domain goes live, pages are set up, and a sitemap.xml file is submitted to Google, Yahoo! and MSN through the new sitemap.org standard. Who picks up the pages first?

Get your bets in for the inaugural index race!

Home | Careers | RSS | Contact Us | Newsletter

International sites: