How ETags and If-Modified-Since headers interact
Part of the fun of writing programs that deal with HTTP is decoding things like RFC 2616 to answer somewhat obscure questions about how various things interact. Today's case is the following question:
When can your web server generate a 304 'content not modified' response if it receives a request with both an
If-Modified-Since are HTTP request headers used
to implement 'conditional GET', a bandwidth saving technique that
avoids re-fetching unchanged pages (see
or here for
more discussion of this).
ETag headers come into this because the server's
ETag value is
what the client will use as its
If-None-Match value in the
conditional GET request.)
You can only generate a 304 response if both headers match; the
If-None-Matchmatches the response's
If-Modified-Sinceheader matches the
In the case of
Last-Modified, servers may
require an exact match instead of merely Last-Modified being no later
than If-Modified-Since. As RFC 2616 notes in
client authors should really just store the Last-Modified result as a
string and hork it up in their If-Modified-Since header.
This came up when I threw debugging code into DWiki to see exactly what various people repeatedly pulling my Atom feed without getting bandwidth-efficient 304 responses were sending. One feed reader was sending both headers but making up their own If-Modified-Since instead of just repeating Last-Modified. (DWiki requires an exact match for technical reasons.)
MSNbot (still) has problems with binary files
Dating back to our first experiences with msnbot, the MSN Search web crawler, I've known that it was kind of crazy about repeatedly fetching large binary files. Since then, we have pointed this issue out to MSN Search people more than once and switched to using accurate Content-Types. Recently we had a week of MSNbot not refetching those large binaries, so it looked like MSNbot had finally been fixed.
So much for that. Since 7pm Wednesday night, MSNbot has fetched 3.1 gigabytes of various large, unchanging 'application/<definitely not text>' files from us. Highlights of the experience include MSNbot fetching fetching the same 537 megabyte ISO image six times (once less than twenty minutes after the previous fetch).
It is clear that MSNbot simply does not deal correctly with binary files, things served with various 'application/<whatever>' content types. There are a few application/* content types that are appropriate to index (PDFs, for example), but for us MSNbot definitely goes far beyond that.
From things I've heard, it would not surprise me if MSNbot ignores the
content-type and just relies on a hard-coded list of URL extensions to
not crawl. (Presumably things like
.zip are in there.)
This is completely brain-damaged, since extensions on URLs don't
necessarily have anything to do with their content-type. For example,
you will search high and low to find a
.html extension in DWiki.
(Yes, some web servers use the file extension as part of the process
to decide on what Content-Type: header to generate. This is an
internal implementation detail.)
I doubt we're the only site experiencing this issue. If you have large binary files on your site, I strongly urge you to check your server logs for similar behavior.