What NFSv3 operations can be in the Linux nfsd reply cache
The Linux kernel NFS server (nfsd) provides a number of statistics
/proc/net/rpc/nfsd, which are often then exposed by metrics
agents such as the Prometheus host agent. One guide to of what
overall information is in this rpc/nfsd file is SvennD's nfsd
stats explained. The
first line is for the "reply cache", which caches replies to NFS
requests so they can be immediately sent back if a duplicate request
comes in. The three numbers provided for the reply cache are cache
hits, cache misses, and the number of requests that aren't cacheable
in the first place. A common explanation of this 'nocache' number is,
well, I will quote the Prometheus host agent's help text for its version
of this metric:
# HELP node_nfsd_reply_cache_nocache_total Total number of NFSd Reply Cache non-idempotent operations (rename/delete/…).
Knowing how many renames, deletes, and so on were going on seemed like a good idea, so I put a graph of this (and some other nfsd RPC numbers) into one of our Grafana dashboards. To my slowly developing surprise, generally almost all of the requests to the NFS servers I was monitoring fell into this 'nocache' category (which was also SvennD's experience, recounted in their entry). So I decided to find out what NFSv3 operations were cacheable and which ones weren't. The answer was surprising.
For NFSv3 operations, the answer is in the big
array at the end of fs/nfsd/nfs3proc.c.
Operations with a
RC_NOCACHE aren't cacheable;
entries with other values are. The non-cacheable NFSv3 operations
access commit fsinfo fsstat getattr lookup null pathconf read readdir readdirplus readlink
The cacheable ones are:
create link mkdir mknod remove rename rmdir setattr symlink write
NFS v3 operations that read information are not cacheable in the reply cache and show up in the 'nocache' category, while NFS v3 operations that change the filesystem mostly are cacheable.
Contrary to what you might guess from the Prometheus host agent's help text and various other sources, the non-cacheable NFS v3 operations aren't things like RENAME and CREATE, they are instead the NFSv3 operations that just read things from the filesystem (with the exception of COMMIT). In particular, GETATTR is an extremely frequent operation, so it's no wonder that most of the time the 'nocache' category dominated in my stats.
If you want to track the number of creates, writes, and so on, what you want to track is the number of misses to the reply cache. Tracking the 'nocache' number tells you how many read operations are happening.
(All of this makes sense once you understand why the reply cache is (or was) necessary, which is for another entry. I actually knew this as background NFS protocol knowledge, but I didn't engage that part of my memory when I was putting together the Grafana graph and had that tempting help text staring me in the face.)