by Head of Search
Andrew Girdwood
Yahoo has trebled del.icio.us' user base after acquiring the social tagging site in December 2005. Could one of the growth strategies have been to continue to allow Googlebot sly access the site?Andrew Girdwood
Del.icio.us is the seminal social search site. It's remarkably simple and to-the-point interface appealed to many of the busy and web-savvy technical gurus who first adopted social tagging. The advert free and uncluttered interface ensured it was easy to use. Yahoo kept the design and pushed the "Save to del.icio.us" feature out across its platform. The tagging's site fraternal twin Flickr was one of the first sites in Yahoo's network and individual photograph pages now each have the option of being bookmarked.
The social tagging site integrates well with Firefox and Internet Explorer. Toolbar additions and bookmarketlets (JavaScript enhanced bookmarks/favourites) allow users to easily add sites to their del.icio.us bookmark list. That is the point of del.icio.us after all, to bookmark sites online and be able to access those bookmarks from any Internet enabled computer anywhere on the world.
The social search aspect is important too. Del.icio.us makes it easy to recommend sites to other people in your social network. The peer interaction is limited but it keeps del.icio.us sticky. Users come back to see what their friends might have found for them.
Yahoo has done well to grow del.icio.us by %300. The search engine had a social search offering of its own - MyWeb - when it bought the site. The technology itself is not a particular challenge, which is why there is a plethora of smaller social tagging sites out there on the World Wide Web but del.icio.us had more to offer Yahoo than code. Yahoo's attraction to del.icio.us was certainly partly due to the established social bookmarking's site user base and brand. It seems unlikely that Yahoo has any plans to merge del.icio.us into MyWeb in the near future.
Social search may be seen as a Web 2.0 concept but traditional search has played a role in del.icio.us' growth. Google drives traffic to the site. That may surprise some. At a glance del.icio.us appears to block Google from the site. A look at the site's robots.txt file (used to restrict where search engine spiders can go) reveals that the site prohibits all search engines from anywhere other than the RSS feed. Del.icio.us' own thumbnail gathering spider is the exception and is allowed the full gambit of the site.
The graphic above depicts the actual file.
Here's the catch. When Googlebot itself requests the same instruction file del.icio.us makes a last minute modification. When Google asks where its spider can go del.icio.us opens up most of the site after all, opting to block only areas like inboxes and log in screens.
Here the graphic depicts the Google-specific robots.txt used by del.icio.us.
Google is not the only search engine to benefit from this careful targeting of instructions. Yahoo's own spider, Yahoo! Slurp, is also allowed in at the last minute.
This is not a technique to be carried out lightly. As a rule of thumb search engines like Google, Yahoo and Windows Live see any attempt to show their search spiders a different version of the page that users see as an attempt to trick them. This trickery, known as cloaking in the search engine optimisation industry, is one of the quickest routes a penalty or even a ban.
There is no suggestion here that del.icio.us is trying to trick Google. It should be remembered that del.icio.us is part of the search engines' technology and therefore likely to be outside of the normal rules.
The likely reason why del.icio.us and Yahoo adopt for this subtle search inclusion is to discourage spam and scraping. Del.icio.us works because the quality of websites in the social bookmarking network tends to be high. It is high because there appears to be no incentive for unethical link builders to target the site. There's little spam.
Scraping may be a problem for del.icio.us too. Scraping is the process whereby an automated agent (a robot, a spider or some script) harvests information such as sites and tag details from del.icio.us for republication elsewhere. It is likely that many of these scraping spiders will disobey the robots.txt file on del.icio.us but the social bookmarking site is certainly incentivized not to make it any easier for these rogue agents.
Scraping and other automated traffic is a growing problem on the internet can lead to vastly inflated page impressions and traffic tallies for websites. Increasingly, sites are turning to other metrics to help calculate their success, ROAS and ROI. The number of subscribed users remains a solid success metric and further reason why a %300+ increase on del.icio.us is good news for Yahoo.
