OpenBSD's 'spinning' CPU time category
Unix systems have long had a basic breakdown of what your CPU (or
CPUs) was spending its time doing. The traditional division is user
time, system time, idle time, and 'nice' time (which is user time
for tasks that have their scheduling priority lowered through
nice(1) or the equivalent),
and then often 'interrupt' time, for how much time the system spent
in interrupt handling. Some Unixes have added 'iowait', which is traditionally defined as 'the
system was idle but one or more processes were waiting for IO to
complete'. OpenBSD doesn't have iowait, but current versions have
a new time category, 'spinning'.
The 'spinning' category was introduced in May of 2018, in this change:
Stopping counting and reporting CPU time spent spinning on a lock as system time.
Introduce a new CP_SPIN "scheduler state" and modify userland tools to display the % of timer a CPU spents spinning.
(This is talking about a kernel lock.)
Since this dates from early 2018, I believe it's in everything from
OpenBSD 6.4 onward. It's definitely in OpenBSD 6.6. This new CPU
time category is supported in OpenBSD's versions of
systat, but it is not explicitly broken out by
vmstat; in fact
vmstat's 'sy' time is actually
the sum of OpenBSD 'system', 'interrupt', and 'spinning'. Third
party tools may or may not have been updated to add this new category.
(I don't know why OpenBSD hasn't updated vmstat. Perhaps they
consider its output frozen for some reason, even though it's hiding
information by merging everything together into 'sy'. The vmstat
manpage is not technically lying about what 'sy' is, since all of
true system time, interrupts, and spinning time are forms of time
spent in the kernel, but the breakdown between those three can be
important. And it means that you can't directly compare vmstat
'sy' to the system time you get from
Our experience is that under some loads, it's possible for a current SMP OpenBSD machine to spend quite appreciable amounts of time in this 'spinning' state. Specifically we've seen our dual CPU OpenBSD L2TP server spend roughly 33% of its time this way while people were apparently trying to push data through it as fast as they could go (which didn't actually go all that fast, perhaps because of all of that spinning).
Marking whether or not the current CPU is spinning on a kernel lock is handled in sys/kern/kern_lock.c, modifying a per-CPU scheduler state spc_spinning field. Tracking and accounting for this is handled in sys/kern/kern_clock.c, in handling the 'statistics clock' (look for use of CP_SPIN). User programs find out all of this through sysctl(2), specifically KERN_CPTIME2 and friends. All of OpenBSD's CPU time categories are found in sys/sched.h.
PS: If you're using a program that doesn't currently support the 'spinning' category, you can reverse engineer the spinning value by adding up all of the other ones and looking for what's missing. Normally, you would expect that all of the categories of CPU time add up to more or less 100%; if you have all but one of them, you can work backward to the missing one based on that. This may not be completely precise, but at least it will pick up large gaps.