== On banning web search engines

There are only a few
[[popular search engines|PragmaticSearchEnginePopularity]] at any
time. They are doing you a favour by indexing you, because you need
their users far more than they need your content. ~~You are are doing
everyone else a favour by letting them index you~~, since they need
content (yours included) to attract users.

When people are doing you a favour, it's worthwhile to go out of your
way to help them out.

When you are doing search engines the favour, it's *their* job to go
out of their way to not be a burden, not yours. Given that we're
already doing unpopular search engines a favour to start with, my
patience for doing any more work to accommodate their special needs is
about nil.

I still have a spirit of neighborliness and thus I'm inclined to let
web spiders rove freely.  But I'm also a
[[pragmatist|../sysadmin/OnBlockingIPRanges]]; when a web spider
starts making us notice it, I want to know what we're getting out of
it. Web spiders that want to keep crawling here had better have a
decent answer.

(It is surprising how few web spiders have web pages that try to
explain why I would want to let them crawl us.)