It's clearly somebody with a poor or intermittent internet connection trying to get reference material available offline, who's deliberately kept their requests to one per second so as to be certain that they're not causing anyone any problems.

In the unlikely event that I noticed this on the websites I administer (in between the flood of bogus connections trying to exploit SSL bugs), I can't imagine myself being even slightly annoyed, so I'm curious as to why this makes you so angry.

To be clear, I'm not angry, just vaguely irritated, and there's a few reasons for that. The first is that despite not having any real impact on the system, this was a drastic increase in traffic; those 72,000 requests by themselves were a bit over three times the size of the usual daily volume. And they were almost all useless requests and thus useless traffic, which simply requested tens of thousands of variations of index pages.

Another is that anyone who wants to get all of the content here can do so very easily if they pay even the littlest bit of attention. There is a 'Full index of entries' link in the sidebar; point your crawler at that page, tell it to follow one level of links, and you're almost completely done. With a little more work you'll have all of the comments as well. If someone is interested in the contents of Wandering Thoughts but can't be bothered to go to even that level of effort and attention, I am not very interested in helping give them a ton of index pages. People deploying HTTrack against here are very likely to be doing so blindly, so.

(I also have long-standing views that web spiders had better respect nofollow, cf. HTTrack is a web spider (and explicitly checks robots.txt), but violates this.)

Yeah, that makes sense.

By the way, not that you have any reason to care, but it amused me for a few minutes: I tried accessing https://utcc.utoronto.ca/ in Windows 98. The result is that IE 5 (which it comes with out of the box) throws the expected SSL error, but Opera 9.64 (the last version that officially supported Win 98) works just fine.

So now you know :-).

To be fair to the HTTrack developers, there are an annoying number of lazy admins which will ban well-behaved scrapers simply because it's easier to set over-broad robots.txt rules and string-match honest User Agents than to analyze logs for bad behaviour ...even if (like me), you go above and beyond (writing a custom scraper) to ensure you download only what you need and to cache more heavily (only ever one HTTP request for a given URL, even if that causes staleness issues) and download fewer supplementary resources (none) than real browsers.

In one case (Fanfiction.net), they are so anti-bot that, for a long time, they set a blanket robots.txt ban up to and including GoogleBot.

I went so far as to write a Python helper module to automate the process of harvesting identifying headers like User-Agent from the default browser on the system and waiting a random amount of time between requests (currently an implementation of how the wget manpage describes --random-wait, but it's more about abstracting it with an API so I'm free to "real humans aren't that random" statistical analysis attacks if necessary.

(If they want to ban me for automating drudge-work like polling image board threads for updates once a day or generating an LRF eBook from a fanfic before I run out the door, they can try to pick me out among the much less polite behaviour of actual web browsers. Hell, if they target my bots for being too polite, I'll start downloading CSS, scripts, images, and fonts that will be discarded in my HTML post-processing, just to muddy the waters.)

I'm sure you will disagree with me as you have a strong opinion on nofollow. but I disagree with you I will explain why.

The universal accepted way to forbid exploration by automated program is robots.txt.

By trying to use nofollow to control it, you try to game the system.

First it was intended to use only by search engine, not general scrapper.

Second it was intended to avoid use of spam on blog and wiki for SEO purpose, never for controlling robots.

Third you can only control nofollow on your own page, so your method has intrinsically a flaw because other pages can link to it.

The fundamental reason you chose to game the system is that you chose an url scheme not compatible with robots.txt which preexisted at the creation of your site. Moreover, you could switch your url mapping to a robots compatible mapping with proper redirection to keep working existent url. I really feel your attitude like I'm right and the whole external world is wrong.