Why do you ban Googlebot in the first place?

Oh, Mark has plenty of words to say on that subject. ^_^

http://www.tamba2.org.uk/T2/archives/2005/03/23/google-steals/
http://www.tamba2.org.uk/T2/archives/2005/03/27/more-on-google/
http://www.tamba2.org.uk/T2/archives/2005/04/24/google-screws/

I have pointed this post out to them on the day I made the post. I have heard nothing from them.

I think you should let them know. They do respond fairly quickly.

[...]

The standard says we should obey the first applicable rule, whereas Googlebot obeys the longest (that is, the most specific) applicable rule.

Now that’s just plain idiotic. Googlebot should obey any rules set to “User-agent: Googlebot” and any rules set to “User-agent: *”, not just the longest. Basically, Google’s help pages say two things:

1. “Use robots.txt to block the Googlebot.”

2. “The Googlebot will not obey the standard syntax of robots.txt file.”

This remind me of the average American:

1. The average American will not vote for the President that’s best for him/her.

2. He/she will vote for the President with the biggest smile.

I’m going to look again.

I think (s)he’s correct.

Googlebot is taking the longest rule applicable it can find.

So, basically, Googlebot sees:

User-agent: Googlebot
Disallow: /

which is very specific but two lines long, and then it finds this:

User-agent: *
Disallow: /gallery
Disallow: /games
Disallow: /images
Disallow: /nota
Disallow: /stats
Disallow: /upb
Disallow: /getout.php

And that rule includes Googlebot and is way longer than the first one, so it’s picked…
At least that’s what I think… :wink:

The standard says we should obey the first applicable rule, whereas Googlebot obeys the longest (that is, the most specific) applicable rule.

In my robots.txt, the longest does not apply, the second is as specific as you can get.
Robots.txt is created to allow flexibility and that is an option I wish to exercise. MSNBot is welcome here, Googlebot is not.

I want the Google person to tell me how to exclude their bot because even though I am following what i think are the rules, Googlebot is disobeying them.
If I have not heard a decent reply in a few days I will post this over at Webmasterworld and other such forums to both publicise it and get more information.

Fact is that even when Google did crawl my site they refuse to return me in results. They’ve taken 100meg of my data - go search for ‘tamba2′ - you will not find a single direct link. Not one. Hardly fair is it ?

and I think that keeps googlebot at bay.