Once upon a time, it was possible to believe that most web spiders hitting your site were broadly beneficial to the (open) web and to people in general. Oh, sure, there were always bad ones (including spammers scraping the web for addresses to spam), but you could at least believe that bad or selfish spiders were the exception. It's my view that these days are over and that on the modern web, most spiders crawling your site are parasites.

My criteria for whether something is or isn't a parasite is a bit of a hand wave; to steal some famous words, ultimately I know it when I see it. Broadly and generally, web spiders are parasites when they don't gather information to serve the general public, they don't make the web better by their presence, and they don't even do something that we'd consider generally useful even for a somewhat restricted group of people (such as the people on a chat channel). There are all sorts of parasites, of course; some are actively evil and are trying to do things that will do you harm (such as harvest email addresses to spam), while others are simply selfish.

What's a selfish, parasitic web spider? As an example, there are multiple companies that crawl the web looking for mentions of brands and then sell information about this to the brands and various other interested people. There are 'sentiment analysis' and 'media monitoring' firms that try to crawl your pages and analyze what you say about products; several of them came up recently. There are companies that perhaps maybe might tell you something about the network of links and connections between sites, but you have to register first and perhaps that means you have to pay them money to get anything useful. At one point there were companies trying to gather up web pages so they could sell a plagiarism analysis service to universities and other people. And so on and so forth, at nearly endless length if you actually look at your web server logs and then start investigating.

(The individual parasitic web spiders don't necessarily crawl at high volume, although some of them certainly will try if you let them, but there are a lot of different ones overall. It's somewhat depressing how many of them seem to be involved in the general Internet ad business, if you construe it somewhat broadly.)

(I wrote about some of my own attitudes on this long ago, in The limits of web spider tolerance. Things have not gotten better since then.)