Comments on: Google respond … pt#1

By: brand cialis

brand cialis — Wed, 30 May 2007 11:31:19 +0000

By: Mark

Mark — Mon, 04 Jul 2005 08:47:09 +0000

I’m actually going back to my original problem.
I banned Google because they were crawling my site but not returning me in any results. Google now have exactly the same access as MSNBot and Inktomi – so I wait to see if they return me too. Position in results in unimportant – just being there would make a change.

By: Angsuman Chakraborty

Angsuman Chakraborty — Mon, 04 Jul 2005 08:21:53 +0000

Did you try changing the order of your rules in robots.txt?

By: MacManX.com » Blogroll Dive: 7/4/05

MacManX.com » Blogroll Dive: 7/4/05 — Mon, 04 Jul 2005 07:59:36 +0000

[...] Owen offers some advice on keeping WordPress v1.5.x up to date via SVN. Michael Heilemann prefers to store his ideas via an archaic analog system. Khaled releases Rin v1.1. Michael Hampton foresees the end of free speech. Jon switches to WordPress. Orson discusses the importance of understanding animals. Angsuman releases an automated WordPress v1.5.1.2 to v1.5.1.3 patch upgrade. Mark debunks yet another asinine statement about the U.S. military. Tom reports that business blogging “more than pays for itself.” And, Mark receives a confusing response from Google. [...]

By: Mark

Mark — Sat, 02 Jul 2005 08:47:53 +0000

James – yea, it was readable.
Matt – I’m pondering what to do:

1 – Do as you say then watch the stats closely to check it is obeyed
2 – Allow g-bot back in and see if I figure in any results.

By: Matt

Matt — Sat, 02 Jul 2005 06:24:24 +0000

So would the solution be to unban them in your .htaccess (which will always be incomplete, they have hundreds of thousands of servers) and just wait until they catch up with the robots.txt?

By: MacManX

MacManX — Fri, 01 Jul 2005 16:48:58 +0000

Is your robots.txt file CHMOD 644?

By: Mark

Mark — Fri, 01 Jul 2005 16:48:05 +0000

From Google:

Thank you for your reply. Once again, our robots do obey robots.txt files. However, our robots only check for a robots.txt file once per day when crawling your site. It’s possible that our robots had not yet noted your robots.txt file when you saw the accesses on your server.

If you’d like to continue preventing our robots from crawling your site, you can continue to include the following in your robots.txt file:

User-agent: Googlebot
Disallow: /

Okaaaay…..but that was already there. So how did it grab 100 megabytes of data ?

By: Mark

Mark — Fri, 01 Jul 2005 14:40:31 +0000

I read it as they’ve seen the 403 and “they interpret this to mean that the site is safe to crawl” and THEN they see the robots.txt which they also ignore – after all, if googlebot is banned, it gives the competition an edge.

And that page ? Nah … I was playing with anonymity programs ;)

By: Gregory Wild-Smith

Gregory Wild-Smith — Fri, 01 Jul 2005 14:35:34 +0000

I thought their point was tho that as you’ve banned them from your site (with a 403) they can’t read the robots.txt to know it’s disallowed? Or did I mis-interpret?

And yeah… they taken action when they feel motivated to. You are totally right: people control code.

re that image: interesting, very interesting. I assume you double checked for spyware? Just in case they were correct?