Unmounting recoverable stale NFS mounts on Linux

July 14, 2014

Suppose that you have NFS mounts go stale on your Linux clients by accident; perhaps you have disabled sharing of some filesystem on the fileserver without quite unmounting it on all the clients first. Now you try to unmount them on the clients and you get the cheerful error message:

# umount /cs/dtr
/cs/dtr was not found in /proc/mounts

You have two problems here. The first problem is in umount.nfs and it is producing the error message you see here. This error message happens because at least some versions of the kernel helpfully change the /proc/mounts output for a NFS mount that they have detected as stale. Instead of the normal output, you get:

fs8:/cs/4/dtr /cs/dtr\040(deleted) nfs rw,nosuid,....

(Instead of plain '/cs/dtr' as the mount point.)

This of course does not match what is in /etc/mtab and umount.nfs errors out with the above error message. As far as I can tell from our brief experience with this today, there is no way to cause the kernel to reverse its addition of this '\040(deleted)' marker. You can make the filesystem non-stale (by re-exporting it or the like), have IO to the NFS mount on the client work fine, and the kernel will still keep it there. You are screwed. To get around this you need to build a patched version of nfs-utils (see also). You want to modify utils/mount/nfsumount.c; search for the error message to find where.

(Note that compiling nfs-utils only gets you a mount.nfs binary. This is actually the same program as umount.nfs; it check its name when invoked to decide what to do, so you need to get it invoked under the right name in some way.)

Unfortunately you're not out of the woods because as far as I can tell many versions of the Linux kernel absolutely refuse to let you unmount a stale NFS mountpoint. The actual umount() system calls will fail with ESTALE even when you can persuade or patch umount.nfs to make them. As far as I know the only way to recover from this is to somehow make the NFS mount non-stale; at this point a patched umount.nfs can make a umount() system call that will succeed. Otherwise you get to reboot the NFS client.

(I have tested and both the umount() and umount2() system calls will fail.)

The kernel side of this problem has apparently been fixed in 3.12 via this commit (found via here), so on really modern distributions such as Ubuntu 14.04 all you may need to do is build a patched umount.nfs. It is very much an issue on older ones such as Ubuntu 12.04 (and perhaps even CentOS 7, although maybe this fix got backported). In the mean time try not to let your NFS mounts become stale, or at least don't let the client kernels notice that they are stale.

(If an NFS mount is stale on the server but nothing on the client has noticed yet, you can still successfully unmount it without problems. But the first df or whatever that gets an ESTALE back from the server blows everything up.)

For additional information on this see eg Ubuntu bug 974374 or Fedora bug 980088 or this linux-nfs mailing list message and thread.

Written on 14 July 2014.
« An obvious reminder: disks can and do die abruptly
A data point on how rapidly spammers pick up addresses from the web »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Jul 14 22:26:37 2014
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.