== Unmounting recoverable stale NFS mounts on Linux

Suppose that you have NFS mounts go stale on your Linux clients by
accident; perhaps you have disabled sharing of some filesystem on
the fileserver without quite unmounting it on all the clients first.
Now you try to unmount them on the clients and you get the cheerful
error message:

.pn prewrap on
>  # umount /cs/dtr
>  /cs/dtr was not found in /proc/mounts

You have two problems here. The first problem is in _umount.nfs_
and it is producing the error message you see here.  This error
message happens because at least some versions of the kernel helpfully
change the _/proc/mounts_ output for a NFS mount that they have
detected as stale. Instead of the normal output, you get:

>  fs8:/cs/4/dtr /cs/dtr\040(deleted) nfs rw,nosuid,....

(Instead of plain '_/cs/dtr_' as the mount point.)

This of course does not match what is in _/etc/mtab_ and _umount.nfs_
errors out with the above error message. As far as I can tell from
our brief experience with this today, there is no way to cause the
kernel to reverse its addition of this '_\040(deleted)_' marker.
You can make the filesystem non-stale (by re-exporting it or the
like), have IO to the NFS mount on the client work fine, and the
kernel will still keep it there. You are screwed. To get around
this you need to build a patched version of [[nfs-utils
http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=summary]] ([[see
also http://linux-nfs.org/wiki/index.php/Main_Page]]). You want
to modify _utils/mount/nfsumount.c_; search for the error message
to find where.

(Note that compiling nfs-utils only gets you a _mount.nfs_ binary.
This is actually the same program as _umount.nfs_; it check its
name when invoked to decide what to do, so you need to get it
invoked under the right name in some way.)

Unfortunately you're not out of the woods because as far as I can
tell many versions of the Linux kernel absolutely refuse to let you
unmount a stale NFS mountpoint. The actual _umount()_ system calls
will fail with _ESTALE_ even when you can persuade or patch
_umount.nfs_ to make them. As far as I know the only way to recover
from this is to somehow make the NFS mount non-stale; at this point
a patched _umount.nfs_ can make a _umount()_ system call that will
succeed.  Otherwise you get to reboot the NFS client.

(I have tested and both the _umount()_ and _umount2()_ system
calls will fail.)

The kernel side of this problem has apparently been fixed in 3.12
via [[this commit
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=8033426e6bdb2690d302872ac1e1fadaec1a5581]]
(found [[via here http://nerdbynature.de/s9y/?297]]), so on really
modern distributions such as Ubuntu 14.04 all you may need to do
is build a patched _umount.nfs_. It is very much an issue on older
ones such as Ubuntu 12.04 (and perhaps even CentOS 7, although maybe
this fix got backported). In the mean time try not to let your NFS
mounts become stale, or at least don't let the client kernels notice
that they are stale.

(If an NFS mount is stale on the server but nothing on the client
has noticed yet, you can still successfully unmount it without
problems. But the first _df_ or whatever that gets an _ESTALE_ back
from the server blows everything up.)

For additional information on this see eg [[Ubuntu bug 974374
https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/974374]]
or [[Fedora bug 980088
https://bugzilla.redhat.com/show_bug.cgi?id=980088]] or [[this
linux-nfs mailing list message and thread
http://www.spinics.net/lists/linux-nfs/msg35393.html]].