What /proc/[pid]/stat's process state means and where it comes from

February 20, 2019

We recently updated to a version of the Prometheus host agent that can report how many of your processes are in various states. Naturally this caused me to look at our systems, where I was surprised to find that we had a bunch of processes in what the host agent called state 'I', which I had never heard of before. This gave me two questions, namely where did the host agent get this state information from, and what did it mean.

The answer to the first question is that the process state being reported comes from /proc/[pid]/stat:

1 (systemd) S [...]

The various fields in this are covered in the proc(5) manpage; the state is the third field. The manpage documents a number of possible values but doesn't include 'I', so clearly there's more.

The authoritative source for these flags is fs/proc/array.c's task_state_array array, and I might as well just quote the current 5.0-rc7 version here directly because it turns out to explain everything very well:

 * The task state array is a strange "bitmap" of
 * reasons to sleep. Thus "running" is zero, and
 * you can test for combinations of others with
 * simple bit tests.
static const char * const task_state_array[] = {

       /* states in TASK_REPORT: */
       "R (running)",          /* 0x00 */
       "S (sleeping)",         /* 0x01 */
       "D (disk sleep)",       /* 0x02 */
       "T (stopped)",          /* 0x04 */
       "t (tracing stop)",     /* 0x08 */
       "X (dead)",             /* 0x10 */
       "Z (zombie)",           /* 0x20 */
       "P (parked)",           /* 0x40 */

       /* states beyond TASK_REPORT: */
       "I (idle)",             /* 0x80 */

(While /proc/[pid]/stat shows only the first letter, the full text shows in /proc/[pid]/status, which may save you a look into the kernel source if Linux someday adds additional states.)

On our machines, the processes that I see in state 'I' tend to be kernel threads (or if you prefer, kernel processes). Having looked at the kernel code, I believe it is probably only possible for kernel threads to be in this state; see the sidebar.

While proc(5) will tell you that processes can only wind up in state 'P' up to kernel 3.13, this is not quite true. Kernel threads associated with offlined CPUs will go into state 'P' on at least the Ubuntu 18.04 4.15 based kernel, and I suspect on any kernel. However, this is not likely to be a common situation.

(We have such a machine for unusual reasons.)

The usual cause for a process being in state 'T' is that it has been SIGSTOP'd (more or less), either due to shell job control suspending it or because someone is using SIGSTOP for other purposes.

One regular case of processes winding up in state 't' is that they're being run under a debugger and the debugger has stopped them (perhaps because they've hit a breakpoint). Given that strace also uses the ptrace(2) system call, I wouldn't be surprised to see strace'd processes also show up in state 't'.

As a side note, inside the kernel the actual task states are much more complicated than this. You can start to see the sausage made in include/linux/sched.h.

Sidebar: What the 'idle' state appears to mean

Based on reading sched.h and various other bits of the kernel source, an 'idle' task is a sleeping uninterruptible kernel thread that is not supposed to contribute to the load average. Normally, processes that are doing uninterruptible sleeps in the kernel contribute to the load average on Linux (although not on all Unixes). I believe that this makes 'I' a state that is currently exclusive to kernel threads and there isn't a directly exposed way of putting a user process into this state.

(You really wouldn't want there to be a direct API for this, because you normally want to be able to interrupt sleeping user processes with things like SIGKILL. When the kernel says 'uninterruptible' here, it really means uninterruptible.)

Written on 20 February 2019.
« The cliffs in the way of adding tests to our Django web app
An advantage of tablets and two-in-one devices over small laptops »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Wed Feb 20 22:46:59 2019
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.