The Slashdot effect is not like regular load

October 26, 2007

So you have a website, and you want it to be able to survive a Slashdotting (these days probably as likely to come from Reddit, Digg, or something I'm too out of touch to have heard of as from Slashdot itself). Assuming that you're not going to blow your bandwidth (a safer assumption than it used to be; these days you can usually offload pictures and videos to someone else), what usually takes sites down in this situation is your web server getting overloaded with the request volume.

Regular, normal load is continuous, long term, involves regular users using your site, and probably broadly distributed across your website (although there are likely to be predictable hot spots). Correspondingly, the only real way to deal with a high regular load, the sort you get from being a popular website, is to make sure that most everything about your website is fast, all the time. This is a hard job (and there's a lot of resources around the web about it).

But the Slashdot effect is not like regular load. It will hit only a few things (an anecdote suggests that almost no one goes beyond the linked-to page), it only lasts for a relatively short time, and most of the visitors are not your regular users. This means that you do not need to be fast all the time; all you really need is to be able to stand up to a temporary load surge against a few pages (and generally only for anonymous users). Doing this is a lot easier than making all of your site fast all the time, since you can do things like shed load through less than perfect caching that has compromises that would not be acceptable for regular traffic.

(One compromise is how stale you can let the cache get. Indeed, one common piece of advice for how to deal with this situation on a dynamic site in an emergency is to make a static HTML version of the linked to page and use internal redirects to serve it directly. So what if new comments don't show up until you regenerate the static HTML? In some cases, this is a feature.)

Written on 26 October 2007.
« Long term storage management in the field
Why I am not really interested in hearing blacklist appeals »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri Oct 26 23:23:36 2007
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.