2005-10-24
On banning web search engines
There are only a few popular search engines at any time. They are doing you a favour by indexing you, because you need their users far more than they need your content. You are are doing everyone else a favour by letting them index you, since they need content (yours included) to attract users.
When people are doing you a favour, it's worthwhile to go out of your way to help them out.
When you are doing search engines the favour, it's their job to go out of their way to not be a burden, not yours. Given that we're already doing unpopular search engines a favour to start with, my patience for doing any more work to accommodate their special needs is about nil.
I still have a spirit of neighborliness and thus I'm inclined to let web spiders rove freely. But I'm also a pragmatist; when a web spider starts making us notice it, I want to know what we're getting out of it. Web spiders that want to keep crawling here had better have a decent answer.
(It is surprising how few web spiders have web pages that try to explain why I would want to let them crawl us.)