2014-10-19
Vegeta, a tool for web server stress testing
Standard stress testing tools like siege (or the venerable ab, which
you shouldn't use) are all systems that do N
concurrent requests at once and see how your website stands up to
this. This model is a fine one for putting a consistent load on
your website for a stress test, but it's not actually representative
of how the real world acts. In the real world you generally don't
have, say, 50 clients all trying to repeatedly make and re-make one
request to you as fast as they can; instead you'll have 50 new
clients (and requests) show up every second.
(I wrote about this difference at length back in this old entry.)
Vegeta is a HTTP load and stress testing tool that I stumbled over at some point. What really attracted my attention is that it uses a 'N requests a second' model, instead of the concurrent request model. As a bonus it will also report not just average performance but also on outliers in the form of 90th and 99th percentile outliers. It's written in Go, which some of my readers may find annoying but which I rather like.
I gave it a try recently and, well, it works. It does what it says it does, which means that it's now become my default load and stress testing tool; 'N new requests a second' is a more realistic and thus interesting test than 'N concurrent requests' for my software (especially here, for obvious reasons).
(I may still do N concurrent requests tests as well, but it'll probably mostly be to see if there are issues that come up under some degree of consistent load and if I have any obvious concurrency race problems.)
Note that as with any HTTP stress tester, testing with high load levels may require a fast system (or systems) with plenty of CPUs, memory, and good networking if applicable. And as always you should validate that vegeta is actually delivering the degree of load that it should be, although this is actually reasonably easy to verify for a 'N new request per second' tester.
(Barring errors, N new requests a second over an M second test run
should result in N*M requests made and thus appearing in your server
logs. I suppose the next time I run a test with vegeta I should
verify this myself in my test environment. In my usage so far I
just took it on trust that vegeta was working right, which in
light of my ab experience may be a little bit
optimistic.)
2014-10-08
Simple web application environments and per-request state
One of the big divides in web programming environments (which are somewhat broader than web frameworks) is between environments that only really have per-request state and every new request starts over with a blank slate and environments with state that persists from request to request. CGI is the archetype of per-request state, but PHP is also famous for it. Many more advanced web environments have potential or actual shared state; sometimes this an explicit feature of the environment over simpler ones.
(One example of a persistent state environment is Node and I'd expect the JVM to generally be another one.)
I have nothing in particular against environments with persistent state and sometimes they're clearly needed (or at least very useful) for doing powerful web applications. But I think it's clear that web environments without it are simpler to program and thus are easier to write simple web things in.
Put simply, in an environment with non-persistent state you can be sloppy. You can change things. You can leave things sitting around the global environment. You can be casual about cleaning up bits and pieces. And you know that anything you do will be wiped away at the end of the request and the next one will start from scratch. An environment with persistent state allows you to do some powerful things but you have to be more careful. It's very easy to 'leak' things into the persistent environment and to modify things in a way that unexpectedly changes later requests, and it can also be easy to literally leak memory or other resources that would have been automatically cleaned up in a per-request environment.
(At this point the pure functional programmers are smugly mentioning the evils of mutable state.)
Speaking from personal experience, keeping track of the state you're changing is hard and it's easy to do something you don't realize. DWiki started out running in a purely non-persistent environment; when I also started running it in a semi-persistent one I found any number of little surprises and things I was doing to myself. I suspect I'd find more if I ran it for a long time in a fully persistent environment.
As a side note, there are some relatively obvious overall advantages to building a web application that doesn't require persistent state even if the underlying web environment you're running in supports it. This may make it useful to at least test your application in an environment that explicitly lacks it, just to make sure that everything still works right.