Article Surfing Archive

Yahoo got more pages indexed than Google. And an enormous amount more. It seems that the Yahoo index is more than twice the size of the Google index. Recently this was also announced by Yahoo, even though they generally don't comment on index sizes.

However, a simple search for the word 'the' in both engines shows a huge difference. Where Google comes up with approx. 3,320,000,000 results, Yahoo spits out the huge amount of 10,200,000,000 results.

Ungoogle like, a couple of weeks later Google responded. Not by word, but in silence. An enormous crawl of the net was performed and even though it still says 'Searching 8,168,684,336 web pages' on the google home page, it is obvious that they have indexed a lot more pages.

These new pages actually were ignored by Google before but it seems that the spider rules of google have been changed in order to increase the index. Obviously this could not happen unnoticed. And noticing it we did. Many sites are seeing reduced traffic since Friday, August 19th, 2005.

Considering that Google has special rules for newly indexed pages, (Simply said: no PR so PR is not taken into account when determining the position of the page*) you can see lots of PR0 pages rank high that weren't there before. Some of the sites I monitor show a decrease in traffic of about 20 to 30%. If Google indexed about 20 to 30% more pages then this explains the decrease in traffic. These new pages get found, but the number of searches has not increased. In other words, the amount of traffic Google generates now is spread over 20 to 30% more pages. High ranking sites lose traffic (especially content rich sites) to all these new pages that have been indexed. In a next PR update this lost traffic should be recovered partly I believe, though all these new pages also contain new links so it is a swag (Silly Wild Ass Guess) to predict what will happen in the next PR update.

It is interesting to see how Google now made the fairly commercial decision to respond to Yahoo's increased index. A leader sometimes needs to punch a challenger in the face in order to maintain the respect of the group. They did this, though for now it seems to be just a small punch, which makes me believe we can expect more from Google.

*Controversial, not generally accepted, theory of mine.