Forcefully breaking NFS locks on Linux NFS servers as of Ubuntu 22.04
As I discovered when I first explored /proc/locks,
the Linux NFS server supports two special files in /proc/fs/nfsd
that will get it to break some of the locks it holds, '
unlock_filesystem' (at least in theory). These files aren't
currently documented in nfsd(7); the references
for them are this 2016 linux-nfs message and thread and this
Red Hat document on them. These
appear to have originally been intended for failover situations,
and one sign of this is that their names in fs/nfsd/nfsctl.c
NFSD_FO_UnlockIP' and '
Each file is used by writing something to it. For '
this is straightforward:
# echo /h/281 >unlock_filesystem
When you do this, all of the NFS locks on that filesystem are immediately dropped by the NFS server. Any NFS clients who think they held locks aren't told about this; as far as they know they have the lock. NFS clients that were waiting to get a lock (because the file was already locked) seem to eventually get given their lock. Because existing lock holders get no notification, this is only a safe operation to do if you're confident that there are no real locks on the filesystem on any NFS clients, and any locks you see on the NFS server are stuck NFS locks, where the NFS server thinks some NFS client has the file locked, but the NFS client disagrees.
We've tested doing this on our Ubuntu 22.04 fileservers (both in production and in a testing environment) and it appears to work and not have any unexpected side effects. It turns out that contrary to what I thought in my /proc/locks update for 22.04, the Ubuntu 22.04 lslocks can still mis-parse /proc/locks under some circumstances; this is util-linux issue #1633, which will only be fixed in v2.39 when it gets released. Until then, build from source or bug your distribution to pull in a fix.
(I had forgotten I'd filed issue #1633 last year and it had gotten fixed back then and only re-discovered it while writing this entry.)
I was going to write a number of things about '
it turns out that all I can write about this file is that I can't
get it to do anything. The kernel source code is in conflict between
whether the IP address you write is supposed to be a client IP
address (comments in fs/nfsd/nfsctl.c) or the server's IP address
as seen by clients (comments in fs/lockd/svcsubs.c;
the Red Hat Page talks
about failover in a way that suggests it was originally intended
for failover and to be given a (failover) server IP address. And
in practice on our testing Ubuntu 22.04 NFS fileserver, writing either IP address to '
makes no difference in what /proc/locks says about locks (and how
other NFS clients waiting for locks react).
unlock_ip' worked, it would behave much the same for releasing
locks as rebooting the NFS client,
but without the whole 'reboot' business. Obviously you'd need to
be very sure that the NFS client didn't actually think it had any
NFS locks on the particular NFS server. Unfortunately Linux has no
easy way to send an artificial 'I have rebooted' notification to a
particular NFS server; however, you can use sm-notify(8) on an NFS
client to tell all of the NFS servers that the client talks to that
the client has 'rebooted', which will cause all of them to release
(Temporarily shutting down everything on a NFS client that might try to get a NFS lock may be easier than rebooting it entirely. Also, with enough contortions you could probably make sm-notify(8) send notifications to only a single fileserver, but it's clearly not how sm-notify is intended to be used.)