More notes on Linux's /proc/locks and NFS as of Ubuntu 22.04

April 28, 2023

About a year ago, when we were still running our NFS fileservers on Ubuntu 18.04, I investigated /proc/locks a bit (it's documented in the proc(5) manual page). Since then we've upgraded our fileservers to Ubuntu 22.04 (which uses Ubuntu's '5.15.0' kernel), and there's some things that are a bit different now, especially on NFS servers.

(Update: oops, I forgot to link to the first entry on /proc/locks.)

On our Ubuntu 22.04 NFS servers, two things are different from how they were in 18.04. First, /proc/locks appears to be complete now, in that it shows all current locks held by NFS clients on NFS exported filesystems. Along with this, the process ID in /proc/locks for such NFS client locks is now consistently the PID of the kernel 'lockd' thread. This gives you a /proc/locks that looks like this:

1: POSIX  ADVISORY  WRITE 13602 00:4f:2237553 0 EOF
2: POSIX  ADVISORY  WRITE 13602 00:2e:486322 0 EOF
3: POSIX  ADVISORY  WRITE 13602 00:2e:485496 0 EOF
4: POSIX  ADVISORY  WRITE 13602 00:2e:486562 0 EOF
5: POSIX  ADVISORY  WRITE 13602 00:2e:486315 0 EOF
6: POSIX  ADVISORY  WRITE 13602 00:2e:541938 0 EOF
7: POSIX  ADVISORY  WRITE 13602 00:4a:2602201 0 EOF
8: POSIX  ADVISORY  WRITE 13602 00:2b:7233288 0 EOF
9: POSIX  ADVISORY  WRITE 13602 00:4a:877382 0 EOF
10: POSIX  ADVISORY  WRITE 13602 00:4a:877913 0 EOF
11: FLOCK  ADVISORY  WRITE 9990 00:19:4993 0 EOF

All of those locks except the last one are NFS locks 'held' by the lockd thread. If you use lslocks(8) it shows 'lockd' (and the PID), making it easy to scan for NFS locks. Lslocks is no more able to find out the actual name of the file than it was before, because the kernel 'lockd' thread doesn't have them open and so lslocks can't do its trick of looking in /proc/<pid>/fd for them.

(Your /proc/locks on a 22.04 NFS server is likely to be bigger than it was on 18.04, possibly a lot bigger.)

The Ubuntu 22.04 version of lslocks is not modern enough to be able to list the inode of these locks (which is available in /proc/locks). However a more recent version of util-linux does have such a version of lslocks; support for listing the inode number was added in util-linux 2.38, and it's not that difficult to build your own copy of lslocks on 22.04. The version I built is willing to use the shared libraries from the Ubuntu util-linux package, so you can just pull the built binary out.

(Locally I wrote a cover script that runs our specially built modern lslocks with '-u -o COMMAND,TYPE,MODE,INODE,PATH', because if we're looking into NFS locks on a fileserver the other information usually isn't too useful.)

These two changes make it much easier to diagnose or rule out 'stuck' NFS locks, because now you can reliably see all of the locks that the NFS server does or doesn't hold, and verify if one of them is for the file that just can't be successfully locked on your NFS clients. If you have access to all of the NFS clients that mount a particular filesystem, you can also check to be sure that none of them have a file locked that the server lists as locked by lockd.

(Actually dealing with such a stuck lock is beyond the scope of this entry. There is a traditional brute force option and some other approaches.)

Comments on this page:

From at 2023-04-29 02:18:16:

If you ever plan to parse /proc/locks by hand, beware of one format quirk that seems to be undocumented: If there are processes waiting to acquire a lock, their rows will have a marker that causes all fields to shift by one.

46: FLOCK  ADVISORY  WRITE 1299564 00:1e:16499570 0 EOF
46: -> FLOCK  ADVISORY  WRITE 1299620 00:1e:16499570 0 EOF
47: FLOCK  ADVISORY  WRITE 1994 00:1e:91556 0 EOF

(Here 1299620 is waiting to lock the same file that 1299564 has an exclusive lock on.)

Written on 28 April 2023.
« I can't recommend serious use of an all-in-one local Grafana Loki setup
A crontab related mistake you can make with internal email ratelimits »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri Apr 28 21:51:33 2023
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.