Don't use ab for your web server stress tests (I like siege instead)

March 20, 2013

Like many other people, I sort of automatically reach for the venerable ab Apache program when I want to do some sort of a web server stress test. I've heard that it has flaws and it's not the best program out there, but surely it's good enough for the basics, right?

Well, no, as I found out recently. I don't know exactly why or what's going on, but ab's concurrency option plain doesn't work; you get nowhere near as much concurrency as you asked for and it claims. Due to my concurrency misunderstanding I got to see this first hand and very vividly. When I ran 'ab -c N' against a test DWiki setup, nowhere near as many worker processes got started and used as there should have been (I believe I asked for 50 concurrent requests and saw only 4 worker processes running, which is very wrong). So my message is simple: do not use ab to test anything you care about. That it's there does not make it worthwhile unless you are very sure that it is not quietly doing something odd on you.

On the other hand I can attest that siege works. When I asked it to make N concurrent requests, well, my worker process count shot right up to what it should have been (in the case of high concurrency, every worker process that I allowed). Siege is also capable of hammering on a fast web server so rapidly that it exhausts your machine's normal range of 28,000 or so local TCP ports. On the one hand this is vaguely annoying. On the other hand I can only describe it as a good problem to have, since it means you are serving requests considerably faster than old sockets can expire out of TIME_WAIT.

(Siege is not perfect and I have not conducted either an exhaustive test of web server stress testers or a careful validation of the numbers it reports. Plus, if you really care about this you will want not just averages for things like response speeds but also 90th and 99th percentiles and distributions and so on. You may also want a more sophisticated model than just concurrent connections, one that more closely models the real world behavior of people.)

(This elaborates on a tweet I made a while ago.)

Written on 20 March 2013.
« The wrong way for a framework to lay out projects
The FreeBSD iSCSI initiator is not ready for serious use (as of 9.1) »

Page tools: View Source, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Wed Mar 20 01:07:17 2013
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.