2023-04-28
More notes on Linux's /proc/locks and NFS as of Ubuntu 22.04
About a year ago, when we were still running our NFS fileservers on Ubuntu 18.04, I investigated /proc/locks
a bit (it's documented in the proc(5) manual page). Since then
we've upgraded our fileservers to Ubuntu 22.04 (which uses Ubuntu's
'5.15.0' kernel), and there's some things that are a bit different
now, especially on NFS servers.
(Update: oops, I forgot to link to the first entry on /proc/locks.)
On our Ubuntu 22.04 NFS servers, two things are different from how
they were in 18.04. First, /proc/locks appears to be complete
now, in that it shows all current locks held by NFS clients on NFS
exported filesystems. Along with this, the process ID in /proc/locks
for such NFS client locks is now consistently the PID of the kernel
'lockd' thread. This gives you a /proc/locks that looks like this:
1: POSIX ADVISORY WRITE 13602 00:4f:2237553 0 EOF 2: POSIX ADVISORY WRITE 13602 00:2e:486322 0 EOF 3: POSIX ADVISORY WRITE 13602 00:2e:485496 0 EOF 4: POSIX ADVISORY WRITE 13602 00:2e:486562 0 EOF 5: POSIX ADVISORY WRITE 13602 00:2e:486315 0 EOF 6: POSIX ADVISORY WRITE 13602 00:2e:541938 0 EOF 7: POSIX ADVISORY WRITE 13602 00:4a:2602201 0 EOF 8: POSIX ADVISORY WRITE 13602 00:2b:7233288 0 EOF 9: POSIX ADVISORY WRITE 13602 00:4a:877382 0 EOF 10: POSIX ADVISORY WRITE 13602 00:4a:877913 0 EOF 11: FLOCK ADVISORY WRITE 9990 00:19:4993 0 EOF [...]
All of those locks except the last one are NFS locks 'held' by the lockd thread. If you use lslocks(8) it shows 'lockd' (and the PID), making it easy to scan for NFS locks. Lslocks is no more able to find out the actual name of the file than it was before, because the kernel 'lockd' thread doesn't have them open and so lslocks can't do its trick of looking in /proc/<pid>/fd for them.
(Your /proc/locks on a 22.04 NFS server is likely to be bigger than it was on 18.04, possibly a lot bigger.)
The Ubuntu 22.04 version of lslocks is not modern enough to be able to list the inode of these locks (which is available in /proc/locks). However a more recent version of util-linux does have such a version of lslocks; support for listing the inode number was added in util-linux 2.38, and it's not that difficult to build your own copy of lslocks on 22.04. The version I built is willing to use the shared libraries from the Ubuntu util-linux package, so you can just pull the built binary out.
(Locally I wrote a cover script that runs our specially built modern lslocks with '-u -o COMMAND,TYPE,MODE,INODE,PATH', because if we're looking into NFS locks on a fileserver the other information usually isn't too useful.)
These two changes make it much easier to diagnose or rule out 'stuck' NFS locks, because now you can reliably see all of the locks that the NFS server does or doesn't hold, and verify if one of them is for the file that just can't be successfully locked on your NFS clients. If you have access to all of the NFS clients that mount a particular filesystem, you can also check to be sure that none of them have a file locked that the server lists as locked by lockd.
(Actually dealing with such a stuck lock is beyond the scope of this entry. There is a traditional brute force option and some other approaches.)