2025-04-16
Looking at what NFSv4 clients have locked on a Linux NVS(v4) server
A while ago I wrote an entry about (not) finding which NFSv4
client owns a lock on a Linux NFS(v4) server,
where the best I could do was pick awkwardly through the raw NFS
v4 client information in /proc/fs/nfsd/clients. Recently
I discovered an alternative to doing this by hand, which is the
nfsdclnts
program, and as a result of digging into it and what I was seeing
when I tried it out, I now believe I have a better understanding
of the entire situation (which was previously somewhat confusing).
The basic thing that nfsdclnts will do is list 'locks' and some
information about them with 'nfsdclnts -t lock
', in addition to
listing other state information such as 'open', for open files, and
'deleg', for NFS v4 delegations. The information it lists is somewhat
limited, for example it will list the inode number but not the
filesystem, but on the good side nfsdclnts is a Python program so
you can easily modify it to report any extra information that exists
in the clients/#/states
files. However, this information about
locks is not complete, because of how file level locks appear to
normally manifest in NFS v4 client state.
(The information in the states
files is limited, although it contains
somewhat more than nfsdclnts shows.)
Here is how I understand NFS v4 locking and states
. To start with,
NFS v4 has a feature called delegations where
the NFS v4 server can hand a lot of authority over a file to a NFS
v4 client. When a NFS v4 client accesses a file, the NFS v4 server
likes to give it a delegation if this is possible; it normally will
be if no one else has the file open or active. Once a NFS v4 client
holds a delegation, it can lock the file without involving the NFS
v4 server. At this point, the client's 'states
' file will report
an opaque 'type: deleg' entry for the file (and this entry may or
may not have a filename or instead be what nfsdclnts will report
as 'disconnected dentry').
While a NFS v4 client has the file delegated, if any other NFS v4
client does anything with the file, including simply opening it,
the NFS v4 server will recall the delegation from the original
client. As a result, the original client now has to tell the NFS
v4 server that it has the file locked. At this point a 'type: lock'
entry for the file appears in the first NFS v4 client's states
file. If the first NFS v4 client releases its lock while the second
NFS v4 client is trying to acquire it, the second NFS v4 client
will not have a delegation for the file, so its lock will show up
as an explicit 'type: lock' entry in its states
file.
An additional wrinkle, a NFS v4 client holding a delegation doesn't
immediately release it once all processes have released their locks,
closed the file, and so on. Instead the delegation may linger on
for some time. If another NFS v4 client opens the file during this
time, the first client will lose the delegation but the second NFS
v4 client may not get a delegation from the NFS v4 server, so its
lock will be visible as a 'type: lock' states
file entry.
A third wrinkle is that multiple clients may hold read-only delegations
for a file and have fcntl() read locks on it at once, with each of
them having a 'type: deleg, access: r' entry for it in their states
files. These will only become visible 'type: lock' states
entries
if the clients have to release their delegations.
So putting this all together:
- If there is a 'type: lock' entry for the file in any
states
file (or it's listed in 'nfsdclnts -t lock'), the file is definitely locked by whoever has that entry. - If there are no 'type: deleg' or 'type: lock' entries for the file,
it's definitely not locked; you can also see this by whether
nfsdclnts lists it as having delegations or locks.
- If there are 'type: deleg' entries for the file, it may or may not be locked by the NFS v4 client (or clients) with the delegation. If the delegation is an 'access: w' delegation, you can see if someone actually has the file locked by accessing the file on another NFS v4 client, which will force the NFS v4 server to recall the delegation and expose the lock if there is one.
If the delegation is 'access: r' and might have multiple read-only
locks, you can't force the NFS v4 server to recall the delegation
by merely opening the file read-only (for example with 'cat file
'
or 'less file
'). Instead the server will only recall the delegation
if you open the file read-write. A convenient way to do this is
probably to use 'flock -x <file> -c
/bin/true
', although this does require you to have more permissions
for the file than simply the ability to read it.
Sidebar: Disabling NFS v4 delegations on the server
Based on trawling various places, I believe this is done by writing
a '0' to /proc/sys/fs/leases-enabled
(or the
equivalent 'fs.leases-enabled' sysctl) and then apparently restarting
your NFS v4 server processes. This will disable all user level uses
of fcntl()'s F_SETLEASE and F_GETLEASE as an additional
effect, and I don't know if this will affect any important programs
running on the NFS server itself.
Based on a study of the kernel source code, I believe that you don't
need to restart your NFS v4 server processes if it's sufficient for
the NFS server to stop handing out new delegations but current
delegations can stay until they're dropped.
(There have apparently been some NFS v4 server and client issues with delegations, cf, along with other NFS v4 issues. However, I don't know if the cure winds up being worse than the disease here, or if there's another way to deal with these stateid problems.)