Process states from
/proc/stat's running and blocked numbers
We recently updated to a version of the Prometheus host agent that can report on
how many processes are in various process states.
The host agent has also long reported
node_procs_blocked metrics, which ultimately come from
Naturally, I cross-compared the two different sets of numbers.
To my surprise, in our environment they could be significantly
different from each other. There turn out to be two reasons for
this, one for each
As far as
procs_running goes, it was always higher than the
number of processes that Prometheus reported as being in state
R'. This turns out to be because Prometheus was counting only
processes, because it looks at what appears in
procs_running counts all threads. When you have a multi-threaded
program, only the main process (or thread) shows up directly in
/proc and so has its
/proc/[pid]/stat inspected. Depending on
how the threading in your program is constructed, this can give
you all sorts of running threads but an idle main process.
(This seems to be what happens with Go programs, including the
Prometheus host agent itself. On otherwise idle machines, the
host agent will routinely report no processes in state
anywhere from 5 to 10 threads in
procs_running. On the same
machine, directly '
/proc/stat consistently reports one
process running, presumably the
The difference between
procs_blocked and processes in state
D' is partly this difference between processes and threads, but
they are also measuring slightly different things.
counts threads that are blocked on real disk IO (technically block
IO), while the '
D' process state is really counting processes
that are in an uninterruptible sleep (in the state
TASK_UNINTERRUPTIBLE, with a caveat about '
I' processes from
my earlier entry). Most processes in state '
are waiting on IO in some form, but there are other reasons processes
can wind up in this state.
In particular, processes waiting on NFS IO will be in state '
but not be counted in
procs_blocked. Processes waiting for
NFS IO are part of
%iowait but since they
are not performing actual block IO, they are not counted in
procs_blocked. You can use this to tell why processes (or
threads) are in an IO wait state; if
procs_blocked is high,
they are waiting on block IO, and if they are just in state '
they are waiting for something else.
(I believe that anything that operates at the block IO layer will
show up in
procs_blocked. I suspect that this includes iSCSI,
among other things.)
Since we make a lot of use of NFS and some machines can be waiting on either NFS or local IO (or sometimes both), I suspect that we're going to have uses for this knowledge. It definitely means that we want to show both metrics in our Grafana dashboards.
The modern danger of locales when you combine sort and cron
It never fails. Every time I use 'sort' in a shell script to be run from cron on a modern Linux machine, it blows up in my face because $LANG defaults to some locale that screws up traditional sort order. I need to start all scripts with:
LANG=C; export LANG
(I wound up elaborating on this, because the trap here is not obvious.)
On modern Linux machines, cron runs your cron jobs with the system's
default locale set (as does systemd when running scripts as part
of systemd units), and that locale is almost certainly one that
screws up the traditional sort order (because almost all of them
do), both for
sort and for other things. If you've set your personal shell
environment up appropriately, your scripts work in your environment
and they even work when you
sudo to root, but then you
deploy them to cron and they fail. If you're lucky, they fail loudly.
This is especially clever and dangerous if you're adding a
to a script that didn't previously need it, as I was today. The
sort version of your script worked even from cron; the new
version is only a small change and it works when you run it by
hand, but now it fails when cron runs it. In my case the failure
was silent and I had to notice from side effects.
(Prometheus's Pushgateway very much
dislikes the order that
sort gives you in the en_US.UTF-8
locale. It turns out that my use of
sort here was actually
superstitious in my particular situation, although there are
other closely related ones where I've needed it.)
I should probably make immediately setting either
$LC_COLLATE a standard part of every shell script that I write
or modify. Even if it's not necessary today, it will be someday in
the future, and it's pretty clear that I won't always remember to
add it when it becomes necessary.