If you are in the "thin affiliate" category and you find yourself in a troublesome situation - for example, you utilise a product feed but are banging your head against the wall wondering why all of your website pages aren't indexed.
Well, since here at bigmouthmedia we have a lot of time for affiliates, we hope to be able to offer a little expert insight into how the wonders of spider efficiency can potentially help you, the affiliate, get more from your search engine.
When bigmouthmedia talk about spider efficiency, we mean the process of making the most of the search engine spiders "resource" assigned to crawl your website. These crawls are designed to determine the importance of the page and the site in the crawling cycle of the web and therefore are instrumental in PageRanks.but each site only has a limited amount of crawling resource. So what can you do to improve efficiency?
If-Modified-Since
Widely ignored, the If-Modified-Since header can be used to advise search engines that a page has not changed since the search engine spiders last visit. The crawling resource can then be attributed elsewhere - an ideal tool for large websites that have little crawling resource to play with. Additionally, within the Google webmaster guidelines, Google highlights that websites should make use of this header; making its employment a positive quality signal you could well be sending over any competitors.Sitemap XML
With recent studies released showing that Googlebot can crawl the web more efficiently using a Sitemap XML e.g. reducing canonicalisation, bigmouthmedia believe a Sitemap XML is a must for any affiliate. Creating a full Sitemap XML is not hard task; there is plenty of quality software out there that can help you achieve this. The true value lies within the attributes you can assign to each URL; priority, changefreq and lastmod. These three values, used loosely by search engines, can be used to dictate to a search engine which URLs have a higher crawling importance than others, which URL has changed most recently and how frequently that URL is likely to change. Sitemaps and If-Modified-Since are only two of many ways to influence the crawl of a search engine spider and increase its efficiency. If you are struggling to get all your webpages indexed, don't immediately presume a penalty on the search engines behalf - think more proactively. Make search engine spiders work for you, and not the other way about. Search engines really do want to make the most of your website, so let them. Bigmouthmedia recommends that if you draw up an annotation, similar to the ones used here detailing the relationships between URLs, you will soon see how you can improve your spider efficiency. Remember, your website is your best marketing tool - let it prosper.
