Wandering Thoughts archives

Understanding and sorting out ZFS pool features

Pretty much every filesystem that wants to be around for a long time needs some way to evolve its format, adding new things (and stopping using old ones); ZFS is no exception. In the beginning, the format of ZFS pools (and filesystems) was set by a version number, but this stopped working very well once Sun were no longer the only people evolving ZFS. To handle the situation with multiple people developing different changes to ZFS, ZFS created a system of what are called 'features', where each feature is more or less some change to how ZFS pools work. Most features are officially independent of each other (although they may not be tested independently in practice). All of this is documented today in the zpool-features(7) manual page, which discusses the general system in detail and then lists all of the current features.

(Your local copy of zpool-features(7) may well list fewer features than the latest upstream development version does. For instance, there's a feature for RAID-Z expansion, which only just landed in the development version.)

Each release or version of ZFS supports some set of features, increasing over time. The Ubuntu 22.04 version of ZFS supports more ZFS features than the Ubuntu 18.04 version did, for example. Moving to a new version of ZFS (for example by upgrading your fileservers from Ubuntu 18.04 to 22.04) deliberately doesn't change the features your current ZFS pools have. Only manual action such as 'zpool upgrade -a' will update them to use new features, and you may well hold off on this even though you've updated ZFS versions.

(One reason to hold off is that perhaps you're worried about reverting to your pre-upgrade state. Another reason is just that you haven't gotten around to it. In the old Solaris 10 days, a 'zpool upgrade' of a pool would cause some degree of service interruption, although I don't think that's supposed to happens today.)

In the very old days, 'zpool status -x' would consider available pool format updates to be an 'error' that made a pool worthy of including in its output, which was kind of infuriating. Later, 'zpool status' downgraded this to merely nagging you all the time. Finally, ZFS introduced a pool property where you could specify what features you wanted your pools to have, via compatibility feature sets and setting the 'compatibility' property to a suitable value. If you set the pool's compatibility property to, say, 'openzfs-2.1-linux', and your pool had all of those features, 'zpool status' now won't claim that it's out of date. Unfortunately, 'zpool upgrade' will still report features that it claims can be upgraded to, although any actual upgrade is supposed to be limited to the compatibility features.

As part of these compatibility sets, there are files that list all of the features in each named set, normally found under /usr/share/zfs/compatibility.d. The format of these files is straightforward and can be used with diff to see that, for example, the features that were added between OpenZFS 2.1 for Linux and OpenZFS 2.2 were blake3, block_cloning, head_errlog, vdev_zaps_v2, and zilsaxattr (all of which you can read about in zpool-features(7)). Often there are convenient symbolic links, so you can see the difference in features that were present on Ubuntu 18.04 (where most of our current ZFS pools were created) and that are now available on Ubuntu 22.04 (which we're now running, so we could update pools to have the new features like zstd compression).

Basic information on what features each of your pools don't have enabled yet can be seen with 'zpool upgrade'. Unfortunately there's no convenient way to get this information for a single pool, because 'zpool upgrade POOL' upgrades the pool, not lists not yet enabled features for just that pool. Also, 'zpool upgrade' will list all features, ignoring the constraints of any 'compatibility' property you may have set on the pool. You can use 'zpool status POOL' to see if a specific pool is fully up to date to its compatibility property (if any), but that's all it can tell you; if it says that the pool hasn't enabled all supported features, there's nothing that will readily tell you which compatible features aren't yet enabled while excluding features you've said are incompatible.

(As far as I can see from the code, upgrading a pool's features through 'zpool upgrade' does respect its 'compatibility' setting, as documented. The current 'zpool upgrade' code to list features that aren't enabled doesn't have any code to cross-check them against your 'compatibility', although I think it would be simple to add.)

Pool features are exposed as 'feature@<name>' ZFS pool properties, so you can see a complete list of the features your version of ZFS supports and their state for any particular pool with 'zpool get all POOL' (this comes for free with all other pool properties, so if you want just the features you'll have to throw in a '| grep feature@'). This is the detailed state, so a feature can be 'disabled', 'enabled', or 'active'; however, whether or not the feature is read-only compatible isn't listed. You can check a specific feature's state with, for example, 'zpool get feature@block_cloning', which can be reassuring if there are reports that a particular feature might cause ZFS pool corruption, prompting a new OpenZFS release with the feature disabled in the kernel code.

(The OpenZFS 2.2.1 release prompted my sudden interest in this area, since I run the ZFS development versions, and caused me to realize that I had once again forgotten how to get a full list of pool features and their state. Maybe I'll remember 'zpool get all POOL' this time around.)

PS: ZFS pool features and pool upgrades are a different thing from ZFS filesystem (format) upgrades. Filesystem format upgrades are still version number based, and I believe the last one was done back when Sun was still a going concern.

Sidebar: Some code trivia

Although ZFS features are represented in the pool by name, the current OpenZFS code has a big numbered list of all of the features it knows about, in include/zfeature_common.h. These are the features that, for example, 'zpool upgrade' will tell you that your pool doesn't have enabled. At the moment it appears that there are 41 of them (cf).

According to comments in module/zfs/zfeature.c, enabling a feature shouldn't have any effect, unlike what happened to us with pool version upgrades back in the Solaris days. This should mean that upgrading a pool is a low-impact operation, since unless you have a very old pool all it's doing is enabling a number of features (many of which may not even become active any time soon, such as zstd compression).