The backend for our recent mirroring
To cope with the visitors to their regular web site, THEMIS
put their ordinary web servers behind eight Squid proxies, which were in turn behind a load balancer
box. This apparently held up very well to the quite a few millions of
extra visitors from Google Mars.
The main movie page was on their
regular site (and thus behind the Squid proxies), but the links to
the movies pointed to
video.mars.asu.edu did was serve up HTTP redirections to the
URLs of the various mirror locations, more or less in rotating through
them to distribute the load. To be as fast and light as possible its web
server didn't bother to look at the HTTP request, so to mirror a second
file the THEMIS people had to run a second server, which they did by
.wmv format movie live on
THEMIS ran an automated monitoring system to detect overloaded or dead
mirrors. It worked by running through the list of mirror URLs every so
HEAD requests to each; if there wasn't a good response
fast enough, that URL got left out of the list used by the redirector
until it came back to life.
Using HTTP redirects meant that the mirroring could be very simple. It didn't need to worry about DNS round robin or having people set up virtual hosts or anything; all it needed was a list of current mirror URLs. (The disadvantage of HTTP redirects is that the mirroring is semi-exposed to your visitors. I don't think THEMIS cared under the circumstances; they were more concerned that demand for the movies would overwhelm ASU's Internet connection.)
Sidebar: why such a primitive HTTP redirector?
Why not parse the HTTP requests on video.mars, instead of having to run two servers and so on? The THEMIS people were concerned that video.mars would see a huge connection rate and wanted it to be as light-weight and reliable as possible. You could build a pretty lightweight solution with something like lighttpd's built in FastCGI gateway going to a local FastCGI server, but it would have had more moving parts and thus been more risky to build on the spot.