Wandering Thoughts archives

What can keep a ZFS pool busy and prevent it from exporting

Suppose that you have some ZFS-based NFS fileservers, perhaps in some sort of failover configuration, and that you want to export some pools from one. So you do:

# zpool export tank
cannot unmount '/x/y': Device busy

The usual set of tools don't show any local processes using the filesystem (not that there really are any, the server being purely an NFS server), and there's no actual NFS activity. In short: rather puzzling. Also annoying, since this stops the pool from exporting.

(This can also happen with 'zfs upgrade -a', which apparently unmounts and remounts the filesystems as it upgrades them.)

When this has happened to us, the thing holding the filesystem busy turned out to be the Solaris NFS lock manager. Temporarily disabling it in svcadm allowed the pool to export without problems:

# svcadm disable svc:/network/nfs/nlockmgr
# zpool export tank
# svcadm enable svc:/network/nfs/nlockmgr

I suspect that this will lose any locks attached to the filesystem, and of course it has side effects if you're still allowing NFS traffic to filesystems in other pools (we weren't).

(My theory is that the NFS lock manager holds filesystems busy when it has active locks against something on the filesystem. Unfortunately I don't know of any way of inspecting its state, although I'm sure there is one. (And using the kernel debugger seems a bit overkill.))

Having written all of this, I checked the fine manual and I see that 'zpool export' has a -f option, and I don't think we tried that when we ran into this problem. However, I feel better knowing what the actual cause is (and 'zfs upgrade' has no -f, so we'd have had to deal with the issue sooner or later).