== Another stupid spider mistake

To follow up my [[earlier entry StupidSpiderMistakes]] on this stuff,
I just saw another [[stunned monkey moment ReallyStupidSpiderII]]:

* ~~you can't randomly add a trailing slash to URLs~~ any more than
  you can randomly remove them.
* *especially* when the URL includes a query parameter, because then
  you're changing the query. And that always works really well.

From the pattern of the stealth spider's requests, I think it is adding
the trailing slash on any URL that doesn't end in a filename with an
extension. This is stunningly braindead, as extensions are nothing more
than a hack workaround so webservers don't need real metadata about what
MIME type a file is.

(It also has other problems, like not properly resolving relative URLs
that use '_.._'.)

This stunned monkey moment is brought to you by the idiotic stealth
spider running from 204.11.99.2, which is claimed to belong to a 'Goo
Khim Yeung' as 204.11.99.0/29. (Assuming that the WHOIS information
is accurate, which it isn't always.)