Using a very old ZFS filesystem can give you a kernel panic on Linux

September 28, 2018

I recently wrote an entry saying that how you migrate ZFS filesystems matters, because if you use ZFS's native features for copying filesystems around a great deal of ZFS's internal on-disk data is retained completely intact. If that data has problems, you have just replicated those problems from your old environment to your new one. Unfortunately this turns out not to be a theoretical problem; some people have hit a situation where having used 'zfs send' to copy very old ZFS filesystems on to a modern Linux kernel has now given them a filesystem with files that cause kernel panics when accessed.

The full details are in ZFS on Linux issue #7910. Summarized, it starts with that ZFS has had a number of different ways to store ACLs in ZFS filesystems, including a very old one that was used (we think) relatively early on on Solaris (sufficiently early that it seems to predate ZFS system attributes, which are from 2010). This old ACL format embeds sufficiently small ACLs into its section of ZFS dnodes, where there is room for 72 bytes of it, and the ACL format also has a (byte) size for how many of those 72 bytes are actually used by 'ACEs' (Access Control Entries, apparently).

Some of these very old Solaris ZFS filesystems contain files with old format ACLs that claim that they both use embedded ACLs and their ACLs are longer than 72 bytes. Given that embedded ACLs can only be at most 72 bytes long, this is impossible, but that's what the on-disk data says (one example claims to have a 112 byte set of ACEs). The ZFS code believes this sufficiently to try to copy however many bytes from the fixed size, 72-byte embedded ACE area to an ACL buffer when you try to access a file and ZFS has to check its ACL. Solaris, OmniOS, and old Linux kernels did not notice that this memory copy was copying some number of random bytes at the end (anything over 72 bytes), and so just sailed blithely onward to checking the resulting 'ACL' (which is some amount of real ACL and some amount of random bytes), which apparently generally worked well enough that no one noticed anything wrong. Modern Linux kernels are generally built with a special kernel configuration option that detects this sort of memory over-copying and panics as a safety and security precaution.

(Specifically, this is the FORTIFY_SOURCE kernel configuration option, which was apparently added in kernel 4.13, in September of 2017 or so, per Kees Cook's entry on security things in 4.13. Ubuntu 18.04 LTS has a kernel that is recent enough to contain this option and Ubuntu built their kernel with it turned on, while the normal 16.04 LTS server kernel is far too old to have it. As a result, some people upgrading from 16.04 to 18.04 hit this panic on their existing ZFS filesystems.)

Most people are extremely unlikely to run into this specific problem; it requires a decade-old ZFS filesystem that has some specific and apparently fairly rare malfunctions in it. But it makes an uncomfortably good illustration of how 'zfs send | zfs recv' will faithfully replicate even things that you don't want, and this can cause real problems if you're sufficiently unlucky.

(We're probably going to still use ZFS send streams for our filesystem migrations, even though some of our filesystems are old enough that they have files with these very old format ACLs. We've never actually used ACLs, so hopefully we're not going to have any files with too-large ones.)

PS: I expect that at some point there will be a proposed change to ZFS on Linux that avoids this kernel panic, but since there are a number of open questions about how best to handle this situation, it will probably take a while before one appears.

(This sort of elaborates on a tweet of mine, which I believe I made when I determined that some of our filesystems did have these very old format ACLs.)

Written on 28 September 2018.
« Learning about Go's unaddressable values and slicing
Addressable values in Go (and unaddressable ones too) »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri Sep 28 01:11:20 2018
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.