How we're handling NFS exports for our ZFS on Linux systems
If you have ZFS based NFS fileservers you're normally supposed to
handle setting up NFS export/sharing permissions through ZFS by
setting and updating the
sharenfs property on ZFS filesystems.
ZFS then worries about keeping the system's NFS export permissions
in sync with what (ZFS) filesystems you have mounted, where you
have them mounted, and what their
sharenfs settings are. There
are all sort of convenient aspects of this and it's what we've done
for years on our current fileservers.
Unfortunately this is not an option for us in ZFS on Linux. I sort of covered why in my entry on
sharenfs problem, but I didn't
mention the core issue for us, which is that ZoL's handling of
sharenfs has no support for the Illumos '
root=' option to provide
root access to the NFS filesystem for only certain systems (instead
of all of them).
In that entry I speculated that we'd
embed our NFS export options as a ZFS user property on ZFS filesystems.
This is sort of the intellectually pure option, but we've decided
to take another way. We're going to be managing our NFS export
permissions entirely outside of ZFS, but reusing '
sanfs', our existing
local filesystem management program.
Sanfs's job is to set up and operate filesystems according to our local policies and specifications; it handles things like filesystem quotas and reservations, knows whether the filesystem should be visible on our deliberately restricted web server, and so on. Since its configuration file is the central point that knows about all of our NFS-visible filesystems, we also use it to automatically generate the NFS mount list for our local automounter. The sanfs configuration file is where we specify (Illumos) NFS export options, including any special additions for particular filesystems:
# Global default shareopts nosuid,sec=sys,rw=nfs_ssh,root=nfs_root # Individual filesystems fs3 /h/281 fs3-staff-01 rw+=cks_dev
+= syntax is something that I'm unreasonably happy about;
it exactly captures the change we almost always want to make to NFS
In the Linux ZoL world, the Linux version of
sanfs will still use
the same configuration file and the same format for NFS export
permissions, but instead of just setting the
sharenfs ZFS property
with the final calculated share options, it will convert them over
to Linux NFS export permissions (using some local knowledge and
the general equivalences) and then directly
manage Linux NFS export permissions using an auxiliary script. This
script does two things. First, it writes or updates a per-filesystem
/etc/exports.d that records the current permissions, and
then it pokes
exportfs to update the actual live permissions to
reflect their new state. Among other reasons, recording the state
of things in
/etc/exports.d makes our NFS export permissions
automatically persist over reboots.
(All our NFS exports will use the
mountpoint option, so they're
not active until and unless the ZFS filesystem is mounted.)
One significant part of what makes this work is that we never
actually use any of the convenient things that ZFS's handling of
sharenfs gives you. We always export ZFS filesystems individually,
we never move them around, and we don't export and import pools (at
least not without explicitly unmounting things on clients, for
good reasons). Without a SAN we definitely
can't ever move a pool between physical machines without a lot of
intervention. Basically, once pools and filesystems are created,
they stay there more or less forever.
PS: The Linux version of
sanfs and the current Illumos version
will in fact literally be using the same configuration file, since
we're inevitably going to be operating both at once for a while.
We have terabytes of data to move across in a couple hundred
filesystems, and that's not exactly going to happen fast, especially
when we haven't even finished developing the Linux fileservers.