Process states from
/proc/stat's running and blocked numbers
We recently updated to a version of the Prometheus host agent that can report on
how many processes are in various process states.
The host agent has also long reported
node_procs_blocked metrics, which ultimately come from
Naturally, I cross-compared the two different sets of numbers.
To my surprise, in our environment they could be significantly
different from each other. There turn out to be two reasons for
this, one for each
As far as
procs_running goes, it was always higher than the
number of processes that Prometheus reported as being in state
R'. This turns out to be because Prometheus was counting only
processes, because it looks at what appears in
procs_running counts all threads. When you have a multi-threaded
program, only the main process (or thread) shows up directly in
/proc and so has its
/proc/[pid]/stat inspected. Depending on
how the threading in your program is constructed, this can give
you all sorts of running threads but an idle main process.
(This seems to be what happens with Go programs, including the
Prometheus host agent itself. On otherwise idle machines, the
host agent will routinely report no processes in state
anywhere from 5 to 10 threads in
procs_running. On the same
machine, directly '
/proc/stat consistently reports one
process running, presumably the
The difference between
procs_blocked and processes in state
D' is partly this difference between processes and threads, but
they are also measuring slightly different things.
counts threads that are blocked on real disk IO (technically block
IO), while the '
D' process state is really counting processes
that are in an uninterruptible sleep (in the state
TASK_UNINTERRUPTIBLE, with a caveat about '
I' processes from
my earlier entry). Most processes in state '
are waiting on IO in some form, but there are other reasons processes
can wind up in this state.
In particular, processes waiting on NFS IO will be in state '
but not be counted in
procs_blocked. Processes waiting for
NFS IO are part of
%iowait but since they
are not performing actual block IO, they are not counted in
procs_blocked. You can use this to tell why processes (or
threads) are in an IO wait state; if
procs_blocked is high,
they are waiting on block IO, and if they are just in state '
they are waiting for something else.
(I believe that anything that operates at the block IO layer will
show up in
procs_blocked. I suspect that this includes iSCSI,
among other things.)
Since we make a lot of use of NFS and some machines can be waiting on either NFS or local IO (or sometimes both), I suspect that we're going to have uses for this knowledge. It definitely means that we want to show both metrics in our Grafana dashboards.