Wandering Thoughts archives

Finding which NFSv4 client owns a lock on a Linux NFS(v4) server

A while back I wrote an entry about finding which NFS client owns a lock on a Linux NFS server, which turned out to be specific to NFS v3 (which I really should have seen coming, since it involved NLM and lockd). Finding the NFS v4 client that owns a lock is, depending on your perspective, either simpler or more complex. The simpler bit is that I believe you can do it all in user space; the more complex is that as far as I've been able to dig, you have to.

Our first stop for NFS v4 locks is the NFS v4 information in /proc/locks. When you hold a (POSIX, NFS) lock, you will see an entry that looks like this:

46: POSIX  ADVISORY  READ 527122 00:36:3211286 0 EOF

This may be READ or WRITE, and it might have a byte range instead of being 0 to EOF. The '00:36:3211286' is the filesystem identifier (the '00:36' part, which is in hex) and then the inode number (in decimal, '3211286'). The other number, 527088, is the process ID of what is holding the lock. For a NFS v4 lock, this will always be some nfsd process, where /proc/<pid>/comm will be 'nfsd'. You'll have a number of nfsd processes (threads), and I don't know if it's always the same PID in /proc/locks.

(In addition, read locks can sometimes appear only as DELEG READ entries in /proc/locks, so they look exactly like simple client opens. It's possible to see multiple DELEG entries for the same file, if multiple NFS v4 clients have it open for reading and/or shared locking. If some NFS v4 client then attempts to get an exclusive lock to the file, the /proc/locks entry can change to a POSIX READ lock.)

To find the client (or clients) with the lock, our starting point is /proc/fs/nfsd/clients, which contains one subdirectory for each client. In these subdirectories, the file 'info' tells you what the client's IP is (and the name it gave the server), and 'states' tells you about what things the particular NFS client is accessing in various ways, including locking. Each entry in 'states' has a type, and this type can include 'lock', and in an ideal world all NFS v4 locks would show up as a states entry of this type. Life is not so nice for us, because the state entry for held locks can also be 'type: deleg', and not all 'type: deleg' entries represent held locks, even for a file that is locked.

A typical states entry for a NFS v4 client may look like this:

- 0x...: { type: lock, superblock: "00:36:3211286", filename: "locktest/fred", owner: "lock id:\x..." }

A 'type: lock' entry can appear for either a shared lock or an exclusive one. Alternately a states entry can look like this:

- 0x...: { type: deleg, access: r, superblock: "00:36:3211286", filename: "locktest/fred" }

It's also possible to see both a 'type: deleg' and a 'type: lock' states entries for a file that has been opened and locked only once from a single client.

In all cases, the important thing is the 'superblock:' field, because this is the same value that appears in /proc/locks.

So as far as I can currently tell, the procedure to find the probable owners of NFS v4 locks is that first you go through /proc/locks and accumulate all of the POSIX locks that are owned by a nfsd process, remembering especially their combined filesystem and inode identification. Then you go through the /proc/fs/nfsd/clients states files for all clients, looking for any matching superblock: values for 'type: lock' or 'type: deleg' entries. If you find a 'type: lock' entry, that client definitely has the file locked. If you find a 'type: deleg' entry, the client might have the file locked, especially if it's a shared POSIX READ lock instead of an exclusive WRITE lock; however, the client might merely have the file open.

If you want to see what a given NFS v4 client (might) have locked, you can do this process backward. Read the client's /proc/fs/nfsd/clients states file, record all superblock: values for 'type: lock' or 'type: deleg' entries, and then see if they show up as POSIX locks in /proc/locks. This won't necessarily get all shared locks (which may show up as merely delegations in both the client's states file and in /proc/locks).

(Presumably the information necessary to locate the locking client or clients with more certainty is somewhere in the kernel data structures. However, so far I've been unable to figure it out in the way that I was able to pull out the NFS v3 lock owner information.)

PS: I'm going to be writing a Python tool for our use based on this digging, so I may get to report back later with corrections to this entry. For our purposes we care more about exclusive locks than shared locks, which makes this somewhat easier.

Sidebar: /proc/locks filesystem identifiers

The '00:36' subfield in /proc/locks that identifies the filesystem is the major and minor device numbers of the stat(2) st_dev field for files, directories, and so on on the filesystem. To determine these without stat'ing something on every filesystem, you can look at the third field of every line in /proc/self/mountinfo, with the provisio that /proc/self/mountinfo's field values are in decimal and /proc/locks has it in hex.

(Unfortunately stat(1) doesn't provide the major and minor numbers separately, and its unified reporting doesn't match /proc/locks if the 'minor' number gets large enough.)