Wandering Thoughts archives

Process states from /proc/[pid]/stat versus /proc/stat's running and blocked numbers

We recently updated to a version of the Prometheus host agent that can report on how many processes are in various process states. The host agent has also long reported node_procs_running and node_procs_blocked metrics, which ultimately come from /proc/stat's procs_running and procs_blocked fields. Naturally, I cross-compared the two different sets of numbers. To my surprise, in our environment they could be significantly different from each other. There turn out to be two reasons for this, one for each /proc/stat field.

As far as procs_running goes, it was always higher than the number of processes that Prometheus reported as being in state 'R'. This turns out to be because Prometheus was counting only processes, because it looks at what appears in /proc, while procs_running counts all threads. When you have a multi-threaded program, only the main process (or thread) shows up directly in /proc and so has its /proc/[pid]/stat inspected. Depending on how the threading in your program is constructed, this can give you all sorts of running threads but an idle main process.

(This seems to be what happens with Go programs, including the Prometheus host agent itself. On otherwise idle machines, the host agent will routinely report no processes in state R but anywhere from 5 to 10 threads in procs_running. On the same machine, directly 'cat'ing /proc/stat consistently reports one process running, presumably the cat itself.)

The difference between procs_blocked and processes in state 'D' is partly this difference between processes and threads, but they are also measuring slightly different things. procs_blocked counts threads that are blocked on real disk IO (technically block IO), while the 'D' process state is really counting processes that are in an uninterruptible sleep (in the state TASK_UNINTERRUPTIBLE, with a caveat about 'I' processes from my earlier entry). Most processes in state 'D' are waiting on IO in some form, but there are other reasons processes can wind up in this state.

In particular, processes waiting on NFS IO will be in state 'D' but not be counted in procs_blocked. Processes waiting for NFS IO are part of %iowait but since they are not performing actual block IO, they are not counted in procs_blocked. You can use this to tell why processes (or threads) are in an IO wait state; if procs_blocked is high, they are waiting on block IO, and if they are just in state 'D', they are waiting for something else.

(I believe that anything that operates at the block IO layer will show up in procs_blocked. I suspect that this includes iSCSI, among other things.)

Since we make a lot of use of NFS and some machines can be waiting on either NFS or local IO (or sometimes both), I suspect that we're going to have uses for this knowledge. It definitely means that we want to show both metrics in our Grafana dashboards.

The modern danger of locales when you combine sort and cron

I tweeted:

It never fails. Every time I use 'sort' in a shell script to be run from cron on a modern Linux machine, it blows up in my face because $LANG defaults to some locale that screws up traditional sort order. I need to start all scripts with:

LANG=C; export LANG

(I wound up elaborating on this, because the trap here is not obvious.)

On modern Linux machines, cron runs your cron jobs with the system's default locale set (as does systemd when running scripts as part of systemd units), and that locale is almost certainly one that screws up the traditional sort order (because almost all of them do), both for sort and for other things. If you've set your personal shell environment up appropriately, your scripts work in your environment and they even work when you su or sudo to root, but then you deploy them to cron and they fail. If you're lucky, they fail loudly.

This is especially clever and dangerous if you're adding a sort to a script that didn't previously need it, as I was today. The pre-sort version of your script worked even from cron; the new version is only a small change and it works when you run it by hand, but now it fails when cron runs it. In my case the failure was silent and I had to notice from side effects.

(Prometheus's Pushgateway very much dislikes the order that sort gives you in the en_US.UTF-8 locale. It turns out that my use of sort here was actually superstitious in my particular situation, although there are other closely related ones where I've needed it.)

I should probably make immediately setting either $LANG or $LC_COLLATE a standard part of every shell script that I write or modify. Even if it's not necessary today, it will be someday in the future, and it's pretty clear that I won't always remember to add it when it becomes necessary.