2013-03-20
Don't use ab for your web server stress tests (I like siege instead)
Like many other people, I sort of automatically reach for the venerable
ab Apache program when I want to do some sort of a web server stress
test. I've heard that it has flaws and it's not the best program out
there, but surely it's good enough for the basics, right?
Well, no, as I found out recently. I don't know exactly why or what's
going on, but ab's concurrency option plain doesn't work; you get
nowhere near as much concurrency as you asked for and it claims. Due to
my concurrency misunderstanding
I got to see this first hand and very vividly. When I ran 'ab -c N'
against a test DWiki setup, nowhere near as many worker processes got
started and used as there should have been (I believe I asked for 50
concurrent requests and saw only 4 worker processes running, which is
very wrong). So my message is simple: do not use ab to test anything
you care about. That it's there does not make it worthwhile unless
you are very sure that it is not quietly doing something odd on you.
On the other hand I can attest that siege works. When I asked it to make N
concurrent requests, well, my worker process count shot right up to
what it should have been (in the case of high concurrency, every worker
process that I allowed). Siege is also capable of hammering on a fast
web server so rapidly that it exhausts your machine's normal range of
28,000 or so local TCP ports. On the one hand this is vaguely annoying.
On the other hand I can only describe it as a good problem to have,
since it means you are serving requests considerably faster than old
sockets can expire out of TIME_WAIT.
(Siege is not perfect and I have not conducted either an exhaustive test of web server stress testers or a careful validation of the numbers it reports. Plus, if you really care about this you will want not just averages for things like response speeds but also 90th and 99th percentiles and distributions and so on. You may also want a more sophisticated model than just concurrent connections, one that more closely models the real world behavior of people.)
(This elaborates on a tweet I made a while ago.)