2022-03-17
I need to remember to check for ZFS filesystems being mounted
Over on the Fediverse I said something:
I keep re-learning the ZFS lesson that you want to check not only for the mount point of ZFS filesystems but also that they're actually mounted, since ZFS can easily have un-mounted datasets due to eg replication in progress.
We have a variety of management scripts on our fileservers that do things on 'all ZFS
filesystems on this fileserver' or 'a specific ZFS filesystem if
it's hosted on this fileserver'. Generally they get their list of
ZFS filesystems and their locations by looking at the mountpoint
property (we set an explicit mount location for all of our ZFS
filesystems, instead of using the default locations). Most of the
time this works fine, but every so often one of the scripts has
blown up and we've quietly fixed it to do better.
The problem is that ZFS filesystems can be visible in things like
'zfs list
' and have a mountpoint
property without actually being
mounted. Most of the time all ZFS filesystems with a mountpoint
will actually be mounted, so most of the time the simpler version
works. However, every so often we're moving a filesystem around
with 'zfs send
' and 'zfs receive
', and either an initial
replication of the filesystem sits unmounted on its new home, or
the old version of the now migrated filesystem sits unmounted on
its old fileserver, retained for a while as a safety measure.
It's not hard to fix our scripts, but we have to find them (and
then remember not to make this mistake again when we write new
scripts). This time around I did do a sweep over all of our scripts
looking for use of 'zfs list
' and the 'mountpoint
' property and
so on, and didn't find anything where we (now) weren't also checking
the 'mounted
' property. Hopefully it will stay that way, now that
I've written this entry to remind myself.
Sidebar: Two reasons other filesystems mostly don't have this problem
The obvious reason that other filesystems mostly don't have this
problem is that they sort of don't have a state where they're present
with a mount point assigned but not actually mounted. The less
obvious reason is that most filesystems don't have a separate tool
to list them; instead you look at the output of 'mount
' or some
other way of looking at what filesystems are mounted, and that
obviously excludes filesystems that aren't. You can do the same
with ZFS, but using 'zfs list
' and so on is often more natural.
(With other filesystems, the rough equivalent is to have a 'noauto
'
filesystem in /etc/fstab
that's not currently mounted. If you get
your list of filesystems from fstab, you'll see the same sort of issue.
Of course in practice you mostly don't look at fstab, since it doesn't
reflect the live state of the system. Things in fstab may be unmounted,
and things not in fstab may be mounted