2014-07-14
Unmounting recoverable stale NFS mounts on Linux
Suppose that you have NFS mounts go stale on your Linux clients by accident; perhaps you have disabled sharing of some filesystem on the fileserver without quite unmounting it on all the clients first. Now you try to unmount them on the clients and you get the cheerful error message:
# umount /cs/dtr /cs/dtr was not found in /proc/mounts
You have two problems here. The first problem is in umount.nfs
and it is producing the error message you see here. This error
message happens because at least some versions of the kernel helpfully
change the /proc/mounts output for a NFS mount that they have
detected as stale. Instead of the normal output, you get:
fs8:/cs/4/dtr /cs/dtr\040(deleted) nfs rw,nosuid,....
(Instead of plain '/cs/dtr' as the mount point.)
This of course does not match what is in /etc/mtab and umount.nfs
errors out with the above error message. As far as I can tell from
our brief experience with this today, there is no way to cause the
kernel to reverse its addition of this '\040(deleted)' marker.
You can make the filesystem non-stale (by re-exporting it or the
like), have IO to the NFS mount on the client work fine, and the
kernel will still keep it there. You are screwed. To get around
this you need to build a patched version of nfs-utils (see
also). You want
to modify utils/mount/nfsumount.c; search for the error message
to find where.
(Note that compiling nfs-utils only gets you a mount.nfs binary.
This is actually the same program as umount.nfs; it check its
name when invoked to decide what to do, so you need to get it
invoked under the right name in some way.)
Unfortunately you're not out of the woods because as far as I can
tell many versions of the Linux kernel absolutely refuse to let you
unmount a stale NFS mountpoint. The actual umount() system calls
will fail with ESTALE even when you can persuade or patch
umount.nfs to make them. As far as I know the only way to recover
from this is to somehow make the NFS mount non-stale; at this point
a patched umount.nfs can make a umount() system call that will
succeed. Otherwise you get to reboot the NFS client.
(I have tested and both the umount() and umount2() system
calls will fail.)
The kernel side of this problem has apparently been fixed in 3.12
via this commit
(found via here), so on really
modern distributions such as Ubuntu 14.04 all you may need to do
is build a patched umount.nfs. It is very much an issue on older
ones such as Ubuntu 12.04 (and perhaps even CentOS 7, although maybe
this fix got backported). In the mean time try not to let your NFS
mounts become stale, or at least don't let the client kernels notice
that they are stale.
(If an NFS mount is stale on the server but nothing on the client
has noticed yet, you can still successfully unmount it without
problems. But the first df or whatever that gets an ESTALE back
from the server blows everything up.)
For additional information on this see eg Ubuntu bug 974374 or Fedora bug 980088 or this linux-nfs mailing list message and thread.