== What is in _/proc/self/mountstats_ for NFS mounts: an introduction

As [[I discovered recently NFSIOShowsInIowait]], for several years
Linux kernels have made a huge amount of per-mount NFS performance
statistics visible in _/proc/self/mountstat_. Unfortunately none of
this is documented. Because I have a use for the information and I'm a
glutton for punishment, I'm going to write up what I've found out.
Mountstats contains so much information that this is going to take
several entries.

To start with, let's talk about the overall format of a filesystem
entry. In a relatively recent kernel, this looks like the following
for an NFS v3 mount over TCP (probably a common case):

.pn prewrap on

 device fs1:/cs/mail mounted on /var/mail with fstype nfs statvers=1.0
         opts:   rw,[... many ...]
         age:    11343100
         caps:   caps=0x3fc7,wtmult=512,dtsize=8192,bsize=0,namlen=255
         sec:    flavor=1,pseudoflavor=1
         events: [... numbers ...]
         bytes:  [... numbers ...]
         RPC iostats version: 1.0  p/v: 100003/3 (nfs)
         xprt:   tcp [... numbers ...]
         per-op statistics
                 NULL: 0 0 0 0 0 0 0 0
                 [more operations]

(Some kernel configurations may produce an additional line with
'_fsc:_'. NFS v4 mounts will have a bunch of additional information
that I haven't looked into because we don't have any.)

The entry ends with a blank line (not shown). There are no less than
four different sets of statistics in this; _events:_ for high-level NFS
events, _bytes:_ for actual data being read and written, _xprt:_ for
low level NFS RPC activity, and then relatively detailed statistics for
each NFS protocol operation. If you're hunting performance issues you
may wind up looking at all of them. As is usual for kernel stats, all
of the numbers are 'from the beginning of time' ones and just count up
as things happen; if you want to get per-second statistics or the like
you need to read the file more than once then work out the difference
between your two readings.

Describing most of the fields in _mountstats_ is sufficiently
complicated that it needs separate entries: [[_bytes:_ and high-level
NFS _events:_ NFSMountstatsBytesEvents]], [[_xprt:_ NFS RPC
information NFSMountstatsXprt]], and [[the per-operation statistics
NFSMountstatsNFSOps]].

The _age:_ field is how many seconds this particular NFS mount has been
in existence. You can use this to compute overall per-interval stats
from all of the counters if you're so inclined, although I don't usually
find overall stats very useful (since our activity is invariably very
bursty).

There are currently a number of programs that try to do something with
this firehose of information. Unfortunately I have yet to stumble over
one that gives what I consider useful reports.

PS: You might be wondering why this is _/proc/<PID>/mountstats_
instead of a general file in _/proc_ or whatever. My understanding
is that the reason for this is that modern Linux systems can have
multiple filesystem namespaces and hence you have to talk about a
specific process's view of mounts. It's traditional to use
_/proc/self/mountstats_ because you can always read your own version
of it.

=== Sidebar: where in the kernel this stuff lives

Since there is no documentation, understanding what is being reported
here requires reading the kernel source. In current 3.12.0-rc3 code,
the overall report is produced by _``nfs_show_stats()''_ in
_fs/nfs/super.c_. This is responsible for _events:_ and _bytes:_;
the information that appears in them is discussed in the comments
for ((include/linux/nfs_iostat.h)).

The RPC information is produced by _``rpc_print_iostats()''_ in
_net/sunrpc/stats.c_. The _xptr:_ line is produced by several
different functions in _net/sunrpc/xprtsock.c_ (which one is used
depends on the connection protocol) using information in a structure
that is described in _include/linux/sunrpc/xprt.h_ (note that these
struct fields are *not* printed in order, so you really do need to
read the code). The per-op statistics are produced using information
described in _include/linux/sunrpc/metrics.h_ but again you'll need
to read the source for the order and details.