== The per-NFS-operation data for NFS mounts in _/proc/self/mountstats_

The last big chunk of statistics in [[_mountstats_ NFSMountstatsIndex]]
is a bunch of separate information for each NFS operation. Since the
exact operations are version-specific, the list of operations depends
on the NFS version in use; however, the statistics are the same for
all protocols (and all operations).

A typical example (for NFS v3) looks like this:

.pn prewrap on
>  per-op statistics
>          NULL: 0 0 0 0 0 0 0 0
>       GETATTR: 192686 192688 0 22726724 21580832 2800 79242 83598
>       SETATTR: 10782 10783 0 1699192 1552608 1139 18919 20152
>  [....]

Operations are actually numerical codes and here they're printed in that
numerical order, although with their standard names instead of their
numbers. For each operation the actual numbers are, in order:

# operations: How many requests we've done for this operation.
# transmissions: How many times we've actually *transmitted* a RPC
  request for this operation. As you might have gathered from the
  [[last entry NFSMountstatsXprt]], this can exceed the operation count
  due to timeouts and retries.
# major timeouts: How many times a request has had a major timeout.
  Major timeouts produce 'nfs: server X not responding, still trying'
  messages. You can have timeouts and retries without having major
  timeouts (and my example lines do).
# bytes sent: This includes not just the RPC payload but also the RPC
  headers and so on. It closely matches the on-the-wire size.
# bytes received: As with bytes sent, this is the full size.
# cumulative queue time: How long (in milliseconds) all requests spent
  queued for transmission before they were sent.
# cumulative response time: How long (in milliseconds) it took to get a
  reply back after the request was transmitted. The kernel comments call
  this the RPC {{AB:RTT:Round Trip Time}}.
# cumulative total request time: How long (in milliseconds) all requests
  took from when they were initially queued to when they were completely
  handled. The kernel calls this the RPC execution time.

At a first guess I think you'll want to divide bytes sent by
transmissions but bytes received by operations if you're computing the
average request size. Doing it differently will only matter in the face
of retransmissions due to retries.

As you can see from my example lines, the total request time is a bit
larger than the sum of the queue time and the response time. I don't
know enough about the Linux NFS RPC code to know where this extra time
may come from.

All of this is extremely useful for performance monitoring, especially
if you sum it up across all operations to get a global view of how RPC
for a given mount is performing. Most NFS RPC operations have relatively
obvious names but it's helpful to know that _COMMIT_ is [[part of NFS v3
writes ../solaris/SlowNFSWritesToZFS]]. The other issue is that in NFS
v3 it's possible to set different flags on _WRITE_ operations (some can
be made synchronous) and the per-operation statistics don't break these
out separately. You may thus see wildly different _WRITE_ performance
numbers at different times depending on whether they are synchronous
(and generally slow) or asynchronous.

(In fact I don't think anything in the overall _mountstats_ data lets
you tell these two sorts of _WRITE_s apart, even to the extent of
finding out how much data is being written with synchronous _WRITE_s and
how much with asynchronous ones. This is an unfortunate omission since
asynch writes are likely to be much faster ones.)

PS: there may be other issues waiting with NFS v4 operations. We don't
use NFS v4 here so I've never looked at it.

=== Sidebar: where to find things in the kernel

The code that prints the data is the _``rpc_print_iostats()''_ routine
in _net/sunrpc/stats.c_. The data is explained in comments in
[[_include/linux/sunrpc/metrics.h_ 
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/include/linux/sunrpc/metrics.h]].