So, how does this Googlebot actually work?Rolf Broer, who works at Onetomarket, just released a very interesting article about Googlebot and its crawling capabilities. Rolf created several tests for the so called superior bot of Google to see how it reacts on particular setups of links and pages. Biggest takeaway is that you can use a Google sitemap to boost the crawlrate. But there is more...
Google's statement about the fact that you shouldn't use more than 100 links per page doesn't make it. The test showed that Google will crawl way over 100 links if it has to. Also Matt Cutts statement that "the amount of pages being crawled is roughly proportional to your PageRank" seems to be a tad of course. In 31 days, Googlebot visited about 375.000 pages on a PageRank 0 website. Rolf declares that when the website would have had a PageRank of 1, Googlebot would crawl over 140.000.000.000 pages in 31 days. That simply means that PageRank doesn't matter at all to get your pages crawled. Breadcrumbs, JavaScript & Semantic Intelligence Rolf and his team at Onetomarket did a few more tests with the Googlebot. For instance, they checked how Googlebot handles breadcrumbs and JavaScript, or Semantic Intelligence of links. The outcomes are definitely worth the read. For Dutch readers, read the Dutch version of the story right here. CommentHave a Conversation with Google0 comments  When you first use it, it may seem like something from a sci-fi movie. Confirmed just yesterday, Conversational Search started rolling out on Google.com in the latest ...... read more
Google+ Updates Announced0 comments  Google+ updates were another big topic at the annual Google I/O Conference. The updates have been in effect since Wednesday. In total 41 new features have been added t...... read more
Will Google Announce Spotify Com...0 comments  Rumor has it Google will be debuting a subscription based music service at its annual developer conference, Google I/O, commencing today. The rumors intensified yester...... read more
Paid Subscriptions Now on YouTube0 comments  YouTube has just announced that it is launching a subscription based program where users can pay to access specific channels. The company said the purpose of this prog...... read more
|
Last Comments
EventsUpcoming event
Last event
Bloggers
Latest Videos
  Columns
Tagcloudapp research matt cutts russia internet apple viral youtube social yandex europe website ses microsoft google tools business google maps iphone seo social media blog advertising video interview google earth ppc mobile facebook search streetview privacy a4uexpo search engine strategies indonesia adwords baidu sem google wave ads newsMost Commented
AgendaMost Read
|
   Search |
© 2013 Searchcowboys.com - All Rights Reserved - All views and opinions expressed are those of the authors of Searchcowboys.
All trademarks, slogans, text or logo representation used or referred to in this website are the property of their respective owners. Sitemap
Comments (6)
Great article offering some real insight Dennis. I guess the 100 links per page myth is another case of Google leading us down the path they want us to go when the reality is very different. But then thats what makes it fun!
Di 20 jul 2010, 12:03
@Gareth Rees, yep, it's all part of the game!
Vr 23 jul 2010, 15:06
Funny to read but good to know. Its all part of the game.
Di 3 aug 2010, 18:18
I think this statement "PageRank doesn't matter at all to get your pages crawled." can be very easily clarified from a very simple things which we already know. They are, crawling is the first step designed to a bot, next comes indexing, next ranking. I have just given a brief process about crawlers behavior without going deep into that. So I would like to say that any search bot even before ranking a page should first crawl and index it and later decide its ranking.
Zo 8 aug 2010, 15:20
Googlebot has been a stranger to me but through your post I gained insightful ideas on how it works.
Vr 20 aug 2010, 05:37
Good article, thanks!
Ma 17 okt 2011, 13:05