The per-NFS-operation data for NFS mounts in
The last big chunk of statistics in
is a bunch of separate information for each NFS operation. Since the
exact operations are version-specific, the list of operations depends
on the NFS version in use; however, the statistics are the same for
all protocols (and all operations).
A typical example (for NFS v3) looks like this:
per-op statistics NULL: 0 0 0 0 0 0 0 0 GETATTR: 192686 192688 0 22726724 21580832 2800 79242 83598 SETATTR: 10782 10783 0 1699192 1552608 1139 18919 20152 [....]
Operations are actually numerical codes and here they're printed in that numerical order, although with their standard names instead of their numbers. For each operation the actual numbers are, in order:
- operations: How many requests we've done for this operation.
- transmissions: How many times we've actually transmitted a RPC request for this operation. As you might have gathered from the last entry, this can exceed the operation count due to timeouts and retries.
- major timeouts: How many times a request has had a major timeout. Major timeouts produce 'nfs: server X not responding, still trying' messages. You can have timeouts and retries without having major timeouts (and my example lines do).
- bytes sent: This includes not just the RPC payload but also the RPC headers and so on. It closely matches the on-the-wire size.
- bytes received: As with bytes sent, this is the full size.
- cumulative queue time: How long (in milliseconds) all requests spent queued for transmission before they were sent.
- cumulative response time: How long (in milliseconds) it took to get a reply back after the request was transmitted. The kernel comments call this the RPC RTT.
- cumulative total request time: How long (in milliseconds) all requests took from when they were initially queued to when they were completely handled. The kernel calls this the RPC execution time.
At a first guess I think you'll want to divide bytes sent by transmissions but bytes received by operations if you're computing the average request size. Doing it differently will only matter in the face of retransmissions due to retries.
As you can see from my example lines, the total request time is a bit larger than the sum of the queue time and the response time. I don't know enough about the Linux NFS RPC code to know where this extra time may come from.
All of this is extremely useful for performance monitoring, especially
if you sum it up across all operations to get a global view of how RPC
for a given mount is performing. Most NFS RPC operations have relatively
obvious names but it's helpful to know that
COMMIT is part of NFS v3
writes. The other issue is that in NFS
v3 it's possible to set different flags on
WRITE operations (some can
be made synchronous) and the per-operation statistics don't break these
out separately. You may thus see wildly different
numbers at different times depending on whether they are synchronous
(and generally slow) or asynchronous.
(In fact I don't think anything in the overall
mountstats data lets
you tell these two sorts of
WRITEs apart, even to the extent of
finding out how much data is being written with synchronous
how much with asynchronous ones. This is an unfortunate omission since
asynch writes are likely to be much faster ones.)
PS: there may be other issues waiting with NFS v4 operations. We don't use NFS v4 here so I've never looked at it.
Sidebar: where to find things in the kernel
The code that prints the data is the
net/sunrpc/stats.c. The data is explained in comments in
Comments on this page:Written on 06 October 2013.