If you have ZFS based NFS fileservers you're normally supposed to handle setting up NFS export/sharing permissions through ZFS by setting and updating the sharenfs property on ZFS filesystems. ZFS then worries about keeping the system's NFS export permissions in sync with what (ZFS) filesystems you have mounted, where you have them mounted, and what their sharenfs settings are. There are all sort of convenient aspects of this and it's what we've done for years on our current fileservers. Unfortunately this is not an option for us in ZFS on Linux. I sort of covered why in my entry on ZoL's sharenfs problem, but I didn't mention the core issue for us, which is that ZoL's handling of sharenfs has no support for the Illumos 'root=' option to provide root access to the NFS filesystem for only certain systems (instead of all of them). In that entry I speculated that we'd embed our NFS export options as a ZFS user property on ZFS filesystems. This is sort of the intellectually pure option, but we've decided to take another way. We're going to be managing our NFS export permissions entirely outside of ZFS, but reusing 'sanfs', our existing local filesystem management program.

Sanfs's job is to set up and operate filesystems according to our local policies and specifications; it handles things like filesystem quotas and reservations, knows whether the filesystem should be visible on our deliberately restricted web server, and so on. Since its configuration file is the central point that knows about all of our NFS-visible filesystems, we also use it to automatically generate the NFS mount list for our local automounter. The sanfs configuration file is where we specify (Illumos) NFS export options, including any special additions for particular filesystems:

# Global default
shareopts  nosuid,sec=sys,rw=nfs_ssh,root=nfs_root

# Individual filesystems
fs3 /h/281 fs3-staff-01   rw+=cks_dev

(The += syntax is something that I'm unreasonably happy about; it exactly captures the change we almost always want to make to NFS export permissions.)

In the Linux ZoL world, the Linux version of sanfs will still use the same configuration file and the same format for NFS export permissions, but instead of just setting the sharenfs ZFS property with the final calculated share options, it will convert them over to Linux NFS export permissions (using some local knowledge and the general equivalences) and then directly manage Linux NFS export permissions using an auxiliary script. This script does two things. First, it writes or updates a per-filesystem file in /etc/exports.d that records the current permissions, and then it pokes exportfs to update the actual live permissions to reflect their new state. Among other reasons, recording the state of things in /etc/exports.d makes our NFS export permissions automatically persist over reboots.

(All our NFS exports will use the mountpoint option, so they're not active until and unless the ZFS filesystem is mounted.)

One significant part of what makes this work is that we never actually use any of the convenient things that ZFS's handling of sharenfs gives you. We always export ZFS filesystems individually, we never move them around, and we don't export and import pools (at least not without explicitly unmounting things on clients, for good reasons). Without a SAN we definitely can't ever move a pool between physical machines without a lot of intervention. Basically, once pools and filesystems are created, they stay there more or less forever.

PS: The Linux version of sanfs and the current Illumos version will in fact literally be using the same configuration file, since we're inevitably going to be operating both at once for a while. We have terabytes of data to move across in a couple hundred filesystems, and that's not exactly going to happen fast, especially when we haven't even finished developing the Linux fileservers.