ZFS can apparently start NFS fileservice before boot finishes
Here's something that I was surprised to discover the other day: ZFS can start serving things over NFS before the system is fully up. Unfortunately this can have a bad effect because it's possible for this NFS traffic to cause further ZFS traffic in some circumstances.
Since this sounds unbelievable, let me report what I saw first. As
our problem NFS fileserver rebooted, it
stalled reporting '
Reading ZFS config:'. At the same time, our
iSCSI backends reported a high ongoing write volume to one pool's
set of disks and
snoop on the fileserver could see active NFS
ptree reported that what was running at the time was the
zfs mount -a' that is part of the /system/filesystem/local target.
(I recovered the fileserver from this problem by the simple method
of disconnecting its network interface. This caused
fail to start, but at least the system was up. ZFS commands like
zfs list' stalled during this stage; I didn't think to do a
to capture the actual mounts.)
Although I can't prove it from the source code, I have to assume
zfs mount -a' is enabling NFS access to filesystems as it
mounts them. An alternate explanation is that
had listings for all of the filesystems (ZFS adds them as part of
sharing them over NFS) and this activated NFS service for filesystems
as they appeared. The net effect is about the same.
This is obviously a real issue if you want your system to be fully up and running okay before any NFS fileservice starts. Since apparently some sorts of NFS traffic under some circumstances can stall further ZFS activity, well, this is something you may care about; we certainly do now.
In theory the SMF dependencies say that /network/nfs/server depends on /system/filesystem/local, as well as nlockmgr (which didn't start). In practice, well, how the system actually behaves is the ultimate proof and all I can do is report what I saw. Yes, this is frustrating. That ZFS and SMF together hide so much in black magic is a serious problem that has made me frustrated before. Among other things it means that when something goes odd or wrong you need to be a deep expert to understand what's going on.