Google. Still thieving
14:28 Sunday 5 Jun 05
Yesterday, I removed the googlebot ban from robots.txt as I was updating that file. I just checked my stats
From AWStats:
Googlebot 1872+20 57.26 MB 04 Jun 2005 – 14:41
I was wondering how long it would take for the bot to wander in, and despite it using 57meg of bandwidth, it does not return this domain in any search result at all. That’s bandwidth theft then. Robots.txt reverting :)
More: Google
Comments (1) --
Read (2136)
1
[...] Do you really want google crawling all over your blog ? Stealing your precious bandwidth? please read this http://www.tamba2.org.uk/T2/archives/2005/06/05/google-still-thieving/ Check your logs and see how much google and the like sap up your band width [...]
03:23 Wednesday 19 Oct 05