What NFSv3 operations can be in the Linux nfsd reply cache

April 9, 2021

The Linux kernel NFS server (nfsd) provides a number of statistics in /proc/net/rpc/nfsd, which are often then exposed by metrics agents such as the Prometheus host agent. One guide to of what overall information is in this rpc/nfsd file is SvennD's nfsd stats explained. The first line is for the "reply cache", which caches replies to NFS requests so they can be immediately sent back if a duplicate request comes in. The three numbers provided for the reply cache are cache hits, cache misses, and the number of requests that aren't cacheable in the first place. A common explanation of this 'nocache' number is, well, I will quote the Prometheus host agent's help text for its version of this metric:

# HELP node_nfsd_reply_cache_nocache_total Total number of NFSd Reply Cache non-idempotent operations (rename/delete/…).

Knowing how many renames, deletes, and so on were going on seemed like a good idea, so I put a graph of this (and some other nfsd RPC numbers) into one of our Grafana dashboards. To my slowly developing surprise, generally almost all of the requests to the NFS servers I was monitoring fell into this 'nocache' category (which was also SvennD's experience, recounted in their entry). So I decided to find out what NFSv3 operations were cacheable and which ones weren't. The answer was surprising.

For NFSv3 operations, the answer is in the big nfsd_procedures3 array at the end of fs/nfsd/nfs3proc.c. Operations with a pc_cachetype of RC_NOCACHE aren't cacheable; entries with other values are. The non-cacheable NFSv3 operations are:

access commit fsinfo fsstat getattr lookup null pathconf read readdir readdirplus readlink

The cacheable ones are:

create link mkdir mknod remove rename rmdir setattr symlink write

NFS v3 operations that read information are not cacheable in the reply cache and show up in the 'nocache' category, while NFS v3 operations that change the filesystem mostly are cacheable.

Contrary to what you might guess from the Prometheus host agent's help text and various other sources, the non-cacheable NFS v3 operations aren't things like RENAME and CREATE, they are instead the NFSv3 operations that just read things from the filesystem (with the exception of COMMIT). In particular, GETATTR is an extremely frequent operation, so it's no wonder that most of the time the 'nocache' category dominated in my stats.

If you want to track the number of creates, writes, and so on, what you want to track is the number of misses to the reply cache. Tracking the 'nocache' number tells you how many read operations are happening.

(All of this makes sense once you understand why the reply cache is (or was) necessary, which is for another entry. I actually knew this as background NFS protocol knowledge, but I didn't engage that part of my memory when I was putting together the Grafana graph and had that tempting help text staring me in the face.)

Written on 09 April 2021.
« Rust's rustup tool is surprisingly nice and well behaved
Why NFS servers generally have a 'reply cache' »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri Apr 9 00:43:01 2021
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.