Wandering Thoughts archives

Forcefully breaking NFS locks on Linux NFS servers as of Ubuntu 22.04

As I discovered when I first explored /proc/locks, the Linux NFS server supports two special files in /proc/fs/nfsd that will get it to break some of the locks it holds, 'unlock_ip' and 'unlock_filesystem' (at least in theory). These files aren't currently documented in nfsd(7); the references for them are this 2016 linux-nfs message and thread and this Red Hat document on them. These appear to have originally been intended for failover situations, and one sign of this is that their names in fs/nfsd/nfsctl.c are 'NFSD_FO_UnlockIP' and 'NFSD_FO_UnlockFS'.

Each file is used by writing something to it. For 'unlock_filesystem' this is straightforward:

# echo /h/281 >unlock_filesystem

When you do this, all of the NFS locks on that filesystem are immediately dropped by the NFS server. Any NFS clients who think they held locks aren't told about this; as far as they know they have the lock. NFS clients that were waiting to get a lock (because the file was already locked) seem to eventually get given their lock. Because existing lock holders get no notification, this is only a safe operation to do if you're confident that there are no real locks on the filesystem on any NFS clients, and any locks you see on the NFS server are stuck NFS locks, where the NFS server thinks some NFS client has the file locked, but the NFS client disagrees.

We've tested doing this on our Ubuntu 22.04 fileservers (both in production and in a testing environment) and it appears to work and not have any unexpected side effects. It turns out that contrary to what I thought in my /proc/locks update for 22.04, the Ubuntu 22.04 lslocks can still mis-parse /proc/locks under some circumstances; this is util-linux issue #1633, which will only be fixed in v2.39 when it gets released. Until then, build from source or bug your distribution to pull in a fix.

(I had forgotten I'd filed issue #1633 last year and it had gotten fixed back then and only re-discovered it while writing this entry.)

I was going to write a number of things about 'unlock_ip', but it turns out that all I can write about this file is that I can't get it to do anything. The kernel source code is in conflict between whether the IP address you write is supposed to be a client IP address (comments in fs/nfsd/nfsctl.c) or the server's IP address as seen by clients (comments in fs/lockd/svcsubs.c; the Red Hat Page talks about failover in a way that suggests it was originally intended for failover and to be given a (failover) server IP address. And in practice on our testing Ubuntu 22.04 NFS fileserver, writing either IP address to 'unlock_ip' makes no difference in what /proc/locks says about locks (and how other NFS clients waiting for locks react).

If 'unlock_ip' worked, it would behave much the same for releasing locks as rebooting the NFS client, but without the whole 'reboot' business. Obviously you'd need to be very sure that the NFS client didn't actually think it had any NFS locks on the particular NFS server. Unfortunately Linux has no easy way to send an artificial 'I have rebooted' notification to a particular NFS server; however, you can use sm-notify(8) on an NFS client to tell all of the NFS servers that the client talks to that the client has 'rebooted', which will cause all of them to release their locks.

(Temporarily shutting down everything on a NFS client that might try to get a NFS lock may be easier than rebooting it entirely. Also, with enough contortions you could probably make sm-notify(8) send notifications to only a single fileserver, but it's clearly not how sm-notify is intended to be used.)