by SEO Consultant
R. Falconer
R. Falconer
Over the years, search engines have provided search engine optimisers with a range of tools to help them facilitate the way their websites are crawled and indexed. By offering better methods of communication between webmasters and search engines, both parties can provide a better service to their customers - search engine users.
The most obvious examples of search tools designed to aid this communication are the webmaster tools interfaces offered by the three major search engines: Google, Yahoo and MSN.
Other tools available help webmasters communicate directly with search engines from the website itself: at link level with the rel="nofollow" microformat (AKA the nofollow tag), at page level with the robots meta tag and canonical link element or at site level by using a robots.txt file.
The use of these on-page instruments in SEO has been well documented and debated. In some cases, they've been talked about ad nauseam. Recently, an interesting article by Paul Teitelman talks about identifying nofollow and "juiceless links". It's an in depth (but not definitive) guide on discovering how much value an inbound link has to your site. A certain link might appear beneficial at first, being from a relevant, authoritative site on a page with a high PageRank - but maybe the link has been nofollowed or 302 redirected, or perhaps the page itself has been nofollowed - meaning it has no value for the linked site.
The article is well worth a read and useful to anyone involved in SEO or link building and serves as a reminder that it's easy to forget useful tools when we don't see or use them often - like the X-Robots-Tag.
This little gem is not mentioned in the above article its use could significantly affect the amount of value your site receives from an inbound link.
Rather than being on-page, like rel="nofollow", robots meta or robots.txt, x-robots is held in the http header of a page. X-Robots-Tag can be used for the same directives as the robots meta tag - commands like noindex, nofollow, nosnippet, noarchive and noodp, but instead of using meta such as:
you can put the same information in the http header:
Meaning the http header might look something like this:
Regarding whether a link has value for your site or not, if the page has nofollow in the X-Robots-Tag, it will not be passing any link value to your page. Although it is not widely used, bigmouthmedia has spotted webmasters slipping it into some pretty sneaky places for their own benefit, a practice that any link builder should bear in mind when trying to figure out link value on some websites.
One of the real benefits of X-Robots-Tag is the communication of information about an individual non-HTML file. It can be used on HTML files, but this is often done in robots meta anyway - other files do not have a head section where this information can be conveyed. Incidentally, if both robots meta and X-Robots-Tag are used and they conflict, Google will adhere to the most restrictive one.
Since various files other than HTML are indexed (pdf, swf, images etc.) it is useful to have control over how they are crawled and indexed. Many webmasters use robots.txt to block the crawling of these files to stop them from being indexed. However, robots.txt only contains crawler directives and will only prevent the file itself from being crawled. If the file is linked to enough, search engines will still index the URL. By using an X-Robots-Tag in the header instead of robots.txt, webmasters can stop this. Similarly, rather than disallowing image crawling in robots.txt, X-Robots-Tag can be used.
Google has a good breakdown on the available directives for Robots.txt, Robots Meta Tag and X-Robots-Tag. Google also has some additional directives that are not used by Yahoo or MSN.
WebBug from aman software is a lightweight and useful application for checking http headers and is available for free..
