Your kernel panics in ZFS on Linux probably aren't actual kernel panics

December 29, 2023

Suppose that you have a ZFS based server and one day its kernel messages contain the following:

VERIFY3(sa.sa_magic == SA_MAGIC) failed (1446876386 == 3100762)
PANIC at zfs_quota.c:89:zpl_get_file_info()
Showing stack for process 6711
CPU: 13 PID: 6711 Comm: dp_sync_taskq Tainted: P           O      5.15.0-88-generic #98-Ubuntu
Hardware name: Supermicro Super Server/X11SPH-nCTF, BIOS 2.0 11/29/2017
Call Trace:
 <TASK>
 show_stack+0x52/0x5c
 dump_stack_lvl+0x4a/0x63
 dump_stack+0x10/0x16
 spl_dumpstack+0x29/0x2f [spl]
 spl_panic+0xd1/0xe9 [spl]
 ? dbuf_rele_and_unlock+0x134/0x540 [zfs]
 [...]

Obviously you've hit a ZFS kernel panic, where ZFS handles internal problems in its traditional way, which is to say by panicing and crashing your server. Except that is almost certainly a lie.

Unless you've changed a non-obvious ZFS kernel parameter, your Linux kernel has not actually paniced; ZFS is merely pretending that it has. We can actually see this in the kernel stack trace being shown here, which lists spl_dumpstack() and especially spl_panic(). There's also a comment about this in the source file:

It is often useful to actually have the panic crash the node so you can then get notified of the event, get the crashdump for later analysis and other such goodies.
But we would still default to the current default of not to do that.

Let me be clear: I think this is a terrible choice for almost everyone except ZFS developers themselves. This looks like a kernel panic to non-experts, in that it has 'PANIC' in the message, it dumps very similar information to a Linux kernel OOPS or other 'panic', and so on. However, because it's not an actual panic it won't trigger any kernel settings you've made to force reboots on panics. Instead it will likely leave your ZFS fileserver with a steadily increasing number of ZFS kernel threads hung waiting for locks, and then force you to reboot things by hand when the problems get really bad (probably uncleanly). This can leave you rather puzzled about what's going on and cause unclear system problems for the proverbial some time (we had one fileserver last for over an hour in this state before it became non-functional enough to trigger alerts).

To force ZFS on Linux to actually panic the kernel when ZFS hits one of these internal 'panics', you need to set the SPL module parameter spl_panic_halt to 1. On a live system, this is done with:

echo 1 >/sys/module/spl/parameters/spl_panic_halt

To make this permanent, you'll need to create a suitable .conf file in /etc/modprobe.d, for example:

$ cat /etc/modprobe.d/spl.conf
options spl spl_panic_halt=1

I recommend including some comments about why this is necessary, so in the future you can understand why you have this mysterious setting.

In an ideal world, the text 'PANIC' in these non-panics would be replaced with something less misleading, like 'SPL-PANIC' or 'SPL-HALTING' (unless the system was actually panicing). That would at least make it clear that this was not a regular kernel panic and came from ZFS's SPL component, not the regular kernel. Better would be to change the default of spl_panic_halt or to otherwise align these SPL panics with normal Linux kernel bug handling.

PS: This doesn't have in OpenZFS on FreeBSD, where the FreeBSD version of spl_panic() simply calls vpanic(9) and so triggers FreeBSD's normal kernel panic behavior and infrastructure.

Written on 29 December 2023.
« The various phases of Prometheus Blackbox's HTTP probe
Email addresses are not good 'permanent' identifiers for accounts »

Page tools: View Source, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri Dec 29 21:46:26 2023
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.