== What the Linux ((rcu_nocbs)) kernel argument does (and my Ryzen issues again) It turns out that [[my Linux Ryzen kernel hangs RyzenMachineLinuxHangs]] appear to be a known bug or issue ([[Ubuntu https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1690085]], [[Fedora https://bugzilla.redhat.com/show_bug.cgi?id=1450769]], [[kernel https://bugzilla.kernel.org/show_bug.cgi?id=196683]]); more fortunately, people have found magic incantations that appear to work around the issue. Part of the magic is some kernel command line arguments, usually cited as: > _``rcu_nocbs''=0-~~N~~ ``processor.max_cstate=1''_ (where _~~N~~_ is the number of CPUs you have minus one.) Magic incantations that I don't understand bug me, especially when they seem to be essential to keeping my system from hanging, so I had to go digging. What ((processor.max_cstate)) does is relatively straightforward. As briefly mentioned in [[kernel-parameters.txt https://www.kernel.org/doc/Documentation/admin-guide/kernel-parameters.txt]], it limits the [[C-states http://www.hardwaresecrets.com/everything-you-need-to-know-about-the-cpu-c-states-power-saving-modes/]] ([[also https://en.wikipedia.org/wiki/Advanced_Configuration_and_Power_Interface#Processor_states]], [[also https://software.intel.com/en-us/articles/power-management-states-p-states-c-states-and-package-c-states]]) that Linux will allow processors to go into. Limiting the CPU to C1 at most doesn't allow for much idling and power saving; [[it might be safe to go as far as C5 https://wiki.gentoo.org/wiki/Ryzen#Random_reboots_with_mce_events]], since the usual additional advice is to disable C6 in the BIOS (if your BIOS supports doing this). On the other hand, I don't know if Ryzens do anything between C1 and C6. The ((rcu_nocbs)) parameter is more involved (and mysterious). To more or less understand it, we need to start with [[Read-Copy-Update (RCU) https://lwn.net/Articles/262464/]] (also [[Wikipedia https://en.wikipedia.org/wiki/Read-copy-update]]). To simplify, RCU handles updates to shared data structures by setting up a new version of the data structure, changing a master location to point to it instead of the old version, and then waiting for everyone to have passed a synchronization point where they're guaranteed to be using the new version instead of the old version. At that point you know the old version is unused and you can free it. The Linux kernel's main RCU code handles the RCU algorithm for you but it doesn't know how to free up your data structures. For that it relies on RCU callbacks that you give it; when RCU determines that the old version of your data structure can be disposed of, it will invoke your callback to do this. Normally, RCU callbacks are invoked in interrupt context as part of [[software interrupt (softirq) https://www.kernel.org/doc/htmldocs/kernel-hacking/basics-softirqs.html]] handling. [[Various people didn't like this https://lwn.net/Articles/522262/]] because softirqs preempt whatever happens to be running at the time whenever an appropriate interrupt happens, so people came up with an alternate approach of having these potentially quite time-consuming RCU callbacks handled by regularly scheduled kernel threads instead. This is said to 'offload' RCU callbacks to these threads. Each offloaded CPU gets its own set of RCU offload kernel threads, but these kernel threads can run on any CPU, not just the CPU they're offloading. This is what ((rcu_nocbs)) controls; it's a list of the CPUs in your system that should have their RCU callbacks offloaded to threads. Normally, people use it to fence off a few CPUs from the random interruptions of softirq RCU callbacks. (See [[here https://wiki.linuxfoundation.org/realtime/documentation/technical_details/rcu]] and [[here https://www.kernel.org/doc/Documentation/kernel-per-CPU-kthreads.txt]] for more information and details.) However, the ((rcu_nocbs=0-N)) setting we're using specifies all CPUs, so it shifts *all* RCU callbacks from softirq context during interrupt handling (on whatever specific CPU involved) to kernel threads (on any CPU). As far as I can see, this has two potentially significant effects, given that [[Matt Dillon of DragonFly BSD has reported an issue with _IRETQ_ that completely stops a CPU under some circumstances https://www.phoronix.com/forums/forum/hardware/processors-memory/955368-some-ryzen-linux-users-are-facing-issues-with-heavy-compilation-loads/page7#post955498]]. First, our Ryzen CPUs will spend less time in interrupt handling, possibly much less time, which may narrow any timing window required to hit Matt Dillon's issue. Second, RCU callback processing will continue even if a CPU stops responding to [[IPIs https://en.wikipedia.org/wiki/Inter-processor_interrupt]], although I expect that a CPU not responding to IPIs is going to cause the Linux kernel various other sorts of heartburn. (Unfortunately, Matt Dillon's issue doesn't correspond well with the observed symptoms, where Ryzens hang under Linux not while busy but while idle. My kernel stack backtraces do suggest that at least one CPU is spinning waiting for its IPI to other CPUs to be fully acknowledged, though, so perhaps there is a related problem. Perhaps there are even several problems.)