Timing durations better in Python (most of the time)
As I mentioned in yesterday's entry on timeouts and exceptions, we have some Python programs that check to make sure various things are working, such as that we can log in to our IMAP servers. Since they're generating Prometheus metrics as part of this, one of the metrics they generate is how long it took.
(My feeling is that if you're going to be generating metrics about something, you should always include timing information just on general principles even if you don't have an immediate use for it.)
Until recently, my code generated this duration information in the obvious way:
stime = time.time() do_some_check(....) dur = time.time() - stime
There is nothing significant wrong with this and most of the time it generates completely accurate timing information. However, it has a little drawback, which is that it's using the current time of day (in general, the clock time). If the system's time gets adjusted during the check, the difference between the starting time of day and the ending time of day doesn't accurately reflect the true duration of the check.
A generally better way to measure durations is to use monotonic
time, obtained through
the operating system promises can never go backwards and isn't
affected by changes in the system's clock. Most of the time the two
are going to be the same or very close (sufficiently close that if
you're using Python, you probably don't care about the timing
difference). But sometimes monotonic time will give you a true
answer when clock time will be noticeably off.
The one limitation of monotonic time is that in some environments, monotonic time doesn't advance when the system is suspended, as might happen with a laptop or a virtual machine. Monotonic time is the time as the operating system experiences it, but not necessarily absolute time passing. If you need absolute time passing and you care about the system being suspended, you may have to use clock time and hope for the best.
(In my environment this code is running on a server that should never suspend or pause, so monotonic time is perfect.)
time.monotonic dates back to Python 3.5 or before, so it should
be pretty safe to use everywhere (fortunately for us, Ubuntu 16.04 is
just recent enough to have Python 3.5). People still using Python 2 are
out of luck; consider it another reason to write new code in Python 3