Google has rolled out its new search platform Caffeine in June, but the company has been a little tight lipped about the implementation details. We were told that Caffeine can incrementally update the search index and produce "50% fresher" results, but Google did not go into further details. All of that has changed with two Google search engineers, Daniel Peng and Frank Dabek publishing a paper describing the technology that drives Caffeine, called Percolator.
Up until now Google used a system called MapReduce to build their search index. Many crawlers would gather content from all over the web and feed it into the MapReduce system which then generates the search index in one large batch operation. Given the size of the index (more than 100 million gigabytes) this process can take many days to complete. The delay between a page being crawled and it showing up in the index used to be about 2-3 days on average, with sites being recrawled every one or two weeks. To be able to index breaking news stories Google split the index into layers, with the fastest layer being updated every 10 seconds. But the majority of the web is in the 'slow' layer which means we are stuck with a multi-day latency.
Tagcloudseo maps facebook realtime streetview yandex viral apple sea sem video blogger matt cutts bing mobile baidu funny blog google maps china event browser images google ads indonesia advertising ppc yahoo street view research youtube iphone microsoft search engine twitter ses london search spain