Wandering Thoughts archives

I use unit tests partly to verify that something works in the first place

A long while ago I read Mostly avoid unit tests (via), in which the author starts out with:

Most of the value of a unit test comes when you change the original (tested) code, but not the test, and can run the test to make sure that all is still well. [...]

This is, broadly, not my experience with a lot of the unit tests that I write. A lot of the time I write tests in large part to verify that the code is correct in the first place, not that it's still correct after I change it; in fact, I may never change the code I'm testing after I write it and verify that it works (ie, that it passes all of my tests). Unit tests are often the easiest way to verify that my code is really working, especially if I want to make sure that corner cases work too, errors are handled correctly, and so on.

Not all code needs this or can be easily tested this way (for example if it interacts with the outside world in complex ways), and it very much helps if you test from inside your modules, not outside of them (often you can only directly reach the complex and potentially incorrect code from inside the module). How much I want unit tests also depends on the language I'm writing in. Languages that offer me a REPL give me a way of exploring whether or not my code works without needing to formalize it as written down tests, so in practice I write fewer tests in them.

(Also, after a certain point it's easier to test if some piece of code works by just running the program and exercising it than it is by writing more tests.)

Even when I could explore whether or not my code works through a REPL or through some ad-hoc scaffolding to run it directly, there's a certain amount of value in writing down a table or group of tests. Writing things down encourages me to more fully check the code's range of inputs and possible problems. It's certainly easier to scan a table to see that I haven't considered some case than it is to think back over checks I've already done by hand and realize that I left something out or that I'm not really checking what I thought I was.

(Much of this is probably obvious, and perhaps there are other ways to do it or these are not really proper unit tests. The modern testing terminology sometimes confuses me.)

Timing durations better in Python (most of the time)

As I mentioned in yesterday's entry on timeouts and exceptions, we have some Python programs that check to make sure various things are working, such as that we can log in to our IMAP servers. Since they're generating Prometheus metrics as part of this, one of the metrics they generate is how long it took.

(My feeling is that if you're going to be generating metrics about something, you should always include timing information just on general principles even if you don't have an immediate use for it.)

Until recently, my code generated this duration information in the obvious way:

stime = time.time()
do_some_check(....)
dur = time.time() - stime

There is nothing significant wrong with this and most of the time it generates completely accurate timing information. However, it has a little drawback, which is that it's using the current time of day (in general, the clock time). If the system's time gets adjusted during the check, the difference between the starting time of day and the ending time of day doesn't accurately reflect the true duration of the check.

A generally better way to measure durations is to use monotonic time, obtained through time.monotonic(), which the operating system promises can never go backwards and isn't affected by changes in the system's clock. Most of the time the two are going to be the same or very close (sufficiently close that if you're using Python, you probably don't care about the timing difference). But sometimes monotonic time will give you a true answer when clock time will be noticeably off.

The one limitation of monotonic time is that in some environments, monotonic time doesn't advance when the system is suspended, as might happen with a laptop or a virtual machine. Monotonic time is the time as the operating system experiences it, but not necessarily absolute time passing. If you need absolute time passing and you care about the system being suspended, you may have to use clock time and hope for the best.

(In my environment this code is running on a server that should never suspend or pause, so monotonic time is perfect.)

PS: time.monotonic dates back to Python 3.5 or before, so it should be pretty safe to use everywhere (fortunately for us, Ubuntu 16.04 is just recent enough to have Python 3.5). People still using Python 2 are out of luck; consider it another reason to write new code in Python 3 instead.