So, how does this Googlebot actually work?

Tue 20 July 2010 10:00, Dennis Sievers

Rolf Broer, who works at Onetomarket, just released a very interesting article about Googlebot and its crawling capabilities. Rolf created several tests for the so called superior bot of Google to see how it reacts on particular setups of links and pages. Biggest takeaway is that you can use a Google sitemap to boost the crawlrate. But there is more...

Google crawlrate being tested

Google's statement about the fact that you shouldn't use more than 100 links per page doesn't make it. The test showed that Google will crawl way over 100 links if it has to. Also Matt Cutts statement that "the amount of pages being crawled is roughly proportional to your PageRank" seems to be a tad of course. In 31 days, Googlebot visited about 375.000 pages on a PageRank 0 website. Rolf declares that when the website would have had a PageRank of 1, Googlebot would crawl over pages in 31 days. That simply means that PageRank doesn't matter at all to get your pages crawled.

Breadcrumbs, JavaScript & Semantic Intelligence

Rolf and his team at Onetomarket did a few more tests with the Googlebot. For instance, they checked how Googlebot handles breadcrumbs and JavaScript, or Semantic Intelligence of links. The outcomes are definitely worth the read. For Dutch readers, read the Dutch version of the story right here

  • Comments (11)
  • Google
  • Tell-a-cowboy

Comments (11)


  • Great article offering some real insight Dennis. I guess the 100 links per page myth is another case of Google leading us down the path they want us to go when the reality is very different. But then thats what makes it fun!

    Di 20 jul 2010, 12:03

  • @Gareth Rees, yep, it's all part of the game!

    Vr 23 jul 2010, 15:06

  • Funny to read but good to know. Its all part of the game.

    Di 3 aug 2010, 18:18

  • I think this statement "PageRank doesn't matter at all to get your pages crawled." can be very easily clarified from a very simple things which we already know. They are, crawling is the first step designed to a bot, next comes indexing, next ranking. I have just given a brief process about crawlers behavior without going deep into that. So I would like to say that any search bot even before ranking a page should first crawl and index it and later decide its ranking.

    Zo 8 aug 2010, 15:20

  • Googlebot has been a stranger to me but through your post I gained insightful ideas on how it works.

    Vr 20 aug 2010, 05:37

  • Good article, thanks!

    Ma 17 okt 2011, 13:05

  • Every day Googlebot behaves differently :-/

    Zo 19 okt 2014, 10:21

  • Thanks for sharing, Googlebot kbytes download many of my sites thankfully.

    Zo 19 okt 2014, 10:23

  • LVMH (LVMHF), Neiman Marcus, Macy's (M +0.7%), Tiffany (TIF +0.5%), and other luxury names will do well this holiday season, predicts retail analyst Kristin Bentz. An outlier: Wearable technology will be an on-trend holiday season item, forecasts Bentz. Early adopter Nike (NKE) could have a big Q4.

    Vr 2 jan 2015, 07:19

  • Thanks for sharing, Googlebot kbytes download many of my sites thankfully.

    Vr 10 apr 2015, 10:35

  • Very very nice....

    Vr 10 apr 2015, 10:41


  • HTML is not allowed. URLs are automatically clickable.
    * Email address is not shown

  • Google Nose
  • Flight Search
  • Flight Search
  • Ingress
  • Google Handwrite
  • Using Google search
  • IPv6 launch
  • Not going to miss the internet

Last Comments


Last event


  • J-P De Clerck
    J-P De Clerck

    Profession: Customer-centric digi...

    Company: Conversionation

  • Sam Murray
    Sam Murray

    Profession: Senior Search Consultant

    Company: Verve Search

  • Susie Hood
    Susie Hood

    Profession: Head of Copywriting

    Company: Click Consult / SEO C...

  • Tom Bogaert
    Tom Bogaert

    Profession: Managing Partner

    Company: QueroMedia

Latest Videos



  • Lizette van der Laan
    Social Media Image

    Is it the real you, the witty you, the person who reads the most interesting articles, makes t...


Subscribe to SC Newsletter:

RSS Feed

Are you a bloggerFacebook


© 2015 - All Rights Reserved - All views and opinions expressed are those of the authors of Searchcowboys.

All trademarks, slogans, text or logo representation used or referred to in this website are the property of their respective owners. Sitemap