Google Percolator

Tue 28 September 2010 09:00, Mark Farragher

Google has rolled out its new search platform Caffeine in June, but the company has been a little tight lipped about the implementation details. We were told that Caffeine can incrementally update the search index and produce "50% fresher" results, but Google did not go into further details. All of that has changed with two Google search engineers, Daniel Peng and Frank Dabek publishing a paper describing the technology that drives Caffeine, called Percolator.

Up until now Google used a system called MapReduce to build their search index. Many crawlers would gather content from all over the web and feed it into the MapReduce system which then generates the search index in one large batch operation. Given the size of the index (more than 100 million gigabytes) this process can take many days to complete. The delay between a page being crawled and it showing up in the index used to be about 2-3 days on average, with sites being recrawled every one or two weeks. To be able to index breaking news stories Google split the index into layers, with the fastest layer being updated every 10 seconds. But the majority of the web is in the 'slow' layer which means we are stuck with a multi-day latency.

Percolator changes all that. In the new system pages are still being crawled, but now any changes are written directly to Google's distributed database. Small programs are embedded in the database (called observers) that respond to these changes. When an observer activates it writes directly into the search index and incrementally updates it to reflect the changed page. This triggers other observers to also activate, and after roughly 50 operations the index is up to date.

The new system is a massive performance improvement for Google. Pages are now indexed 100 times faster than before, so if my calculations are correct Google can now process a page in less than an hour! And with this reduction in processing speed there is now room for a lot more pages in the index. Caffeine processes three times as many pages as the old MapReduce system, and the average age of a page in the index is down by 50% which should put it at 1-2 days at most.

So why is this good news for SEO's? Well, you can now run SEO campaigns around breaking news stories using longtail keywords and have a much better chance at getting indexed. A tight deadline used to be a showstopper for SEO, forcing us to resort to AdWords or viral marketing with social media. With Percolator we can use traditional SEO techniques and still produce good results overnight.

... Or so I hope. What do you think?

<Google Instant Launched

Advanced Search Marketing & Web ...>

Comments (3)

- Camuel
- [website]
Obviously Percolator is better for those who use google search and hence make more sense for white-hat SEO

Wed 13 Oct 2010, 01:05
- Camuel Gilyadov
- [website]
See also my blog post on it http://bigdatacraft.com/archives/240

Wed 13 Oct 2010, 01:06
- Samskriti
- [website]
Yup you are right. Keyword rankings are coming faster than earlier and losing the rankings also in the same speed :) To keep the rankings we should have good content and quality back-links.

Mon 1 Nov 2010, 14:42

Comment

Baidu SEO Tips: continue

Fri 4 May 2012, Kelly Ge

0 comments

Followed by my article on Baidu sites – a new tip for Baidu SEO, this time I collected some tips on Baidu SEO and present in brief. For SEO, no matter what kind ...... read more

Baidu sites – a new tip for Baid...

Fri 2 Mar 2012, Kelly Ge

0 comments

You can find many articles about “what is important for Baidu SEO” giving you all kinds of tips. Some tips are used by many search engines, and some are fr...... read more

Baidu heatmap

Fri 20 Jan 2012, Kelly Ge

0 comments

There is nothing special about heatmap. You can easily find one and use it for your site, and many like it, for it is easy to get an idea on browsing behavior of visit...... read more

Baidu can index Weibo (Chinese T...

Fri 9 Dec 2011, Kelly Ge

0 comments

Be careful! Your comments on Weibo can be indexed and searched by others on Baidu. How can it happen? Baidu Open data makes it possible. I have introduced Baidu open d...... read more