OpenBSD's 'spinning' CPU time category

March 27, 2020

Unix systems have long had a basic breakdown of what your CPU (or CPUs) was spending its time doing. The traditional division is user time, system time, idle time, and 'nice' time (which is user time for tasks that have their scheduling priority lowered through nice(1) or the equivalent), and then often 'interrupt' time, for how much time the system spent in interrupt handling. Some Unixes have added 'iowait', which is traditionally defined as 'the system was idle but one or more processes were waiting for IO to complete'. OpenBSD doesn't have iowait, but current versions have a new time category, 'spinning'.

The 'spinning' category was introduced in May of 2018, in this change:

Stopping counting and reporting CPU time spent spinning on a lock as system time.

Introduce a new CP_SPIN "scheduler state" and modify userland tools to display the % of timer a CPU spents spinning.

(This is talking about a kernel lock.)

Since this dates from early 2018, I believe it's in everything from OpenBSD 6.4 onward. It's definitely in OpenBSD 6.6. This new CPU time category is supported in OpenBSD's versions of top and systat, but it is not explicitly broken out by vmstat; in fact vmstat's 'sy' time is actually the sum of OpenBSD 'system', 'interrupt', and 'spinning'. Third party tools may or may not have been updated to add this new category.

(I don't know why OpenBSD hasn't updated vmstat. Perhaps they consider its output frozen for some reason, even though it's hiding information by merging everything together into 'sy'. The vmstat manpage is not technically lying about what 'sy' is, since all of true system time, interrupts, and spinning time are forms of time spent in the kernel, but the breakdown between those three can be important. And it means that you can't directly compare vmstat 'sy' to the system time you get from top or systat.)

Our experience is that under some loads, it's possible for a current SMP OpenBSD machine to spend quite appreciable amounts of time in this 'spinning' state. Specifically we've seen our dual CPU OpenBSD L2TP server spend roughly 33% of its time this way while people were apparently trying to push data through it as fast as they could go (which didn't actually go all that fast, perhaps because of all of that spinning).

Marking whether or not the current CPU is spinning on a kernel lock is handled in sys/kern/kern_lock.c, modifying a per-CPU scheduler state spc_spinning field. Tracking and accounting for this is handled in sys/kern/kern_clock.c, in handling the 'statistics clock' (look for use of CP_SPIN). User programs find out all of this through sysctl(2), specifically KERN_CPTIME2 and friends. All of OpenBSD's CPU time categories are found in sys/sched.h.

PS: If you're using a program that doesn't currently support the 'spinning' category, you can reverse engineer the spinning value by adding up all of the other ones and looking for what's missing. Normally, you would expect that all of the categories of CPU time add up to more or less 100%; if you have all but one of them, you can work backward to the missing one based on that. This may not be completely precise, but at least it will pick up large gaps.

Written on 27 March 2020.
« Any KVM over IP systems need to be on secure networks
The Prometheus host agent's CPU utilization metrics can be a bit weird »

Page tools: View Source.
Search:
Login: Password:

Last modified: Fri Mar 27 00:50:58 2020
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.