== OpenBSD's 'spinning' CPU time category Unix systems have long had a basic breakdown of what your CPU (or CPUs) was spending its time doing. The traditional division is user time, system time, idle time, and 'nice' time (which is user time for tasks that have their scheduling priority lowered through [[_nice(1)_ https://man.openbsd.org/nice.1]] or the equivalent), and then often 'interrupt' time, for how much time the system spent in interrupt handling. Some Unixes have added [['iowait' IowaitAndMultipleCPUs]], which is traditionally defined as 'the system was idle but one or more processes were waiting for IO to complete'. OpenBSD doesn't have iowait, but current versions have a new time category, 'spinning'. The 'spinning' category was introduced in May of 2018, in [[this change https://marc.info/?l=openbsd-cvs&m=152630109526317&w=2]]: > Stopping counting and reporting CPU time spent spinning on a lock as > system time. > > Introduce a new ``CP_SPIN'' "scheduler state" and modify userland > tools to display the % of timer a CPU spents spinning. (This is talking about a kernel lock.) Since this dates from early 2018, I believe it's in everything from OpenBSD 6.4 onward. It's definitely in OpenBSD 6.6. This new CPU time category is supported in OpenBSD's versions of _top_ and _systat_, but it is not explicitly broken out by _vmstat_; in fact [[_vmstat_ https://man.openbsd.org/vmstat]]'s 'sy' time is actually the sum of OpenBSD 'system', 'interrupt', and 'spinning'. Third party tools may or may not have been updated to add this new category. (I don't know why OpenBSD hasn't updated vmstat. Perhaps they consider its output frozen for some reason, even though it's hiding information by merging everything together into 'sy'. The vmstat manpage is not technically lying about what 'sy' is, since all of true system time, interrupts, and spinning time are forms of time spent in the kernel, but the breakdown between those three can be important. And it means that you can't directly compare vmstat 'sy' to the system time you get from _top_ or _systat_.) Our experience is that under some loads, it's possible for a current SMP OpenBSD machine to spend quite appreciable amounts of time in this 'spinning' state. Specifically we've seen our dual CPU OpenBSD [[L2TP https://en.wikipedia.org/wiki/Layer_2_Tunneling_Protocol]] server spend roughly 33% of its time this way while people were apparently trying to push data through it as fast as they could go (which didn't actually go all that fast, perhaps because of all of that spinning). Marking whether or not the current CPU is spinning on a kernel lock is handled in [[sys/kern/kern_lock.c https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/kern/kern_lock.c?rev=1.71&content-type=text/x-cvsweb-markup]], modifying a per-CPU scheduler state ``spc_spinning'' field. Tracking and accounting for this is handled in [[sys/kern/kern_clock.c https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/kern/kern_clock.c?rev=1.101&content-type=text/x-cvsweb-markup]], in handling the 'statistics clock' (look for use of ``CP_SPIN''). User programs find out all of this through [[sysctl(2) https://man.openbsd.org/sysctl.2]], specifically [[KERN_CPTIME2 https://man.openbsd.org/sysctl.2#KERN_CPTIME2_2]] and friends. All of OpenBSD's CPU time categories are found in [[sys/sched.h https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/sys/sched.h?rev=1.56&content-type=text/x-cvsweb-markup]]. PS: If you're using a program that doesn't currently support the 'spinning' category, you can reverse engineer the spinning value by adding up all of the other ones and looking for what's missing. Normally, you would expect that all of the categories of CPU time add up to more or less 100%; if you have all but one of them, you can work backward to the missing one based on that. This may not be completely precise, but at least it will pick up large gaps.