So, how does this Googlebot actually work?

Tue 20 July 2010 10:00, Dennis Sievers

Rolf Broer, who works at Onetomarket, just released a very interesting article about Googlebot and its crawling capabilities. Rolf created several tests for the so called superior bot of Google to see how it reacts on particular setups of links and pages. Biggest takeaway is that you can use a Google sitemap to boost the crawlrate. But there is more...

Google crawlrate being tested

Google's statement about the fact that you shouldn't use more than 100 links per page doesn't make it. The test showed that Google will crawl way over 100 links if it has to. Also Matt Cutts statement that "the amount of pages being crawled is roughly proportional to your PageRank" seems to be a tad of course. In 31 days, Googlebot visited about 375.000 pages on a PageRank 0 website. Rolf declares that when the website would have had a PageRank of 1, Googlebot would crawl over pages in 31 days. That simply means that PageRank doesn't matter at all to get your pages crawled.

Breadcrumbs, JavaScript & Semantic Intelligence

Rolf and his team at Onetomarket did a few more tests with the Googlebot. For instance, they checked how Googlebot handles breadcrumbs and JavaScript, or Semantic Intelligence of links. The outcomes are definitely worth the read. For Dutch readers, read the Dutch version of the story right here

  • Comments (27)
  • Google
  • Tell-a-cowboy

Comments (27)



  • HTML is not allowed. URLs are automatically clickable.
    * Email address is not shown

  • Google Nose
  • Flight Search
  • Flight Search
  • Ingress
  • Google Handwrite
  • Using Google search
  • IPv6 launch
  • Not going to miss the internet

Last Comments


Last event


  • J-P De Clerck
    J-P De Clerck

    Profession: Customer-centric digi...

    Company: Conversionation

  • Sam Murray
    Sam Murray

    Profession: Senior Search Consultant

    Company: Verve Search

  • Susie Hood
    Susie Hood

    Profession: Head of Copywriting

    Company: Click Consult / SEO C...

  • Tom Bogaert
    Tom Bogaert

    Profession: Managing Partner

    Company: QueroMedia

Latest Videos



  • Lizette van der Laan
    Social Media Image

    Is it the real you, the witty you, the person who reads the most interesting articles, makes t...


Subscribe to SC Newsletter:

RSS Feed

Are you a bloggerFacebook


© 2016 - All Rights Reserved - All views and opinions expressed are those of the authors of Searchcowboys.

All trademarks, slogans, text or logo representation used or referred to in this website are the property of their respective owners. Sitemap