Caffeine – Google's new search index

The Google Webmaster Central blog has announced that it has rolled out, across all data centres a new indexing system called Caffeine, which they claim will provide fresher results from a much larger index.

The old Google index consisted of several layers. Re-indexing just one of these layers involved crawling the entire internet. The main layer was being re-indexed every 2 weeks but other layers were crawled much less frequently. When a user performs a Google search they are searching the Google index not the internet, therefore the old indexing system was often days if not weeks out of date.

With Caffeine content is being continuously added to the search index, which means a tweet or a blog post is searchable within seconds of publication. Every second, Caffeine processes hundreds of thousands of web pages, adding them straight into the index. Google claims this makes the search results 50% fresher.

The engineers at Google, headed by Amit Singhal constantly tweak the Google search algorithm making as many as 400 changes throughout a typical year. At the beginning of June, changes to the algorithm saw many sites lose up to 60% of their traffic. Common factors amongst these, included sites with stale content, smaller sites and those with poor quality backlinks. The sites which received a boost in traffic seemed to have all the qualities Google is aiming the Caffeine update at, i.e. fresh content, regular updates and good quality links from relevant sites.

Amit Singhal’s “Search Quality” department is finding that the future of Google Search is to find out what people mean, not what they ask for and then to work out what they will be asking for next. All this has to be done in 112 languages and at a rate of hundreds of millions of queries a day. The new Caffeine indexing system gives the team a great platform to take their plans forward.