== The bytes and events data for NFS mounts in _/proc/self/mountstats_ 

The per NFS mount _mountstats_ performance stats ([[see here for
an introduction NFSMountstatsIndex]]) have two sets of high level
statistics, reported in the _bytes:_ and _events:_ lines. Both of
these come from counters that are described in comments in
[[((include/linux/nfs_iostat.h))
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/include/linux/nfs_iostat.h]]
in the kernel source.
Of the two, the simpler is _bytes:_.

.pn prewrap on

A typical _bytes:_ line looks like:

>  bytes:  2320629391 2297630544 0 0 2298347151 2297630544 718354 717816

In order, let's call these fields nread, nwrite, dread, dwrite, nfsread,
nfswrite, pageread, and pagewrite. These count bytes read and written
to the server with simple _read()_ and _write()_, with _read()_ and
_write()_ calls in ((O_DIRECT)) mode, the actual number of bytes read
and written from the NFS server (regardless of how), and the number of
*pages* (not bytes) read or written via directly _mmap()_'d files. I
believe that the page size is basically always 4 Kb (at least on x86).
It's routine for the ((O_DIRECT)) numbers to be zero.
The most useful numbers of these for performance are what I've
called nfsread and nfswrite, the fifth and sixth fields, because
these represent the actual IO to the server.

A typical _events:_ line looks like this:
>  events: 3717478 126331741 28393 1036981 3355459 1099901 133724160 1975168 3589 2878751 1405861 5669601 720939 96113 3235157 225053 30643 3026061 0 23184 1675425 24 0 0 0 0 0

The _events:_ line tracks various sorts of high level NFS events. There
are a lot of them, so I am just going to list them in order (with field
numbers and some commentary):

# inode revalidate: How many times cached inode attributes have to be
  re-validated from the server.
# dnode revalidate: How many times cached dentry nodes (ie, name to
  inode mappings) have to be re-validated. I suspect that this spawns inode
  revalidations as well.
# data invalidate: How many times an inode had its cached data thrown out.
# attribute invalidate: How many times an inode has had cached inode
  attributes invalidated.

# vfs open: How many times files or directories have been _open()_'d.
# vfs lookup: How many name lookups in directories there have been.
# vfs access: How many times permissions have been checked via the
  internal equivalent of _access()_.
# vfs update page: Count of updates (and potential writes) to pages.
# vfs read page: This is the same as what I called pageread in the
  _bytes:_ field. (Quite literally. The counters are incremented
  next to each other in the source.)
# vfs read pages: Count of how many times a *group* of (mapped?) pages
  have been read. I believe it spawns 'vfs page read' events too but I'm
  not sure.
# vfs write page: Same as pagewrite in _bytes:_.
# vfs write pages: Count of grouped page writes. Probably spawns 'vfs
  write page' events too.
# vfs getdents: How many times directory entries have read with
  _getdents()_. These reads can be served from cache and don't necessarily
  imply actual NFS requests.
# vfs setattr: How many times we've set attributes on inodes.
# vfs flush: How many times pending writes have been forcefully flushed
  to the server (which can happen for various reasons).
# vfs fsync: How many times _fsync()_ has been called on directories (which
  is a no-op for NFS) and files. Sadly you can't tell which is which.
# vfs lock: How many times people have tried to lock (parts of) a file,
  including in ways that are basic errors and will never succeed.
# vfs file release: Basically a count of how many times files have been
  closed and released.

# congestion wait: Not used for anything as far as I can tell.
  There doesn't seem to be anything in the current kernel source that
  actually increments the counter.

# truncation: How many times files have had their size truncated.
# write extension: How many times a file has been grown because you're
  writing beyond the existing end of the file.
# silly rename: How many times you removed a file while it was still
  open by some process, forcing the kernel to instead rename it
  to '_.nfsXXXXXX_' and delete it later.
# short read: The NFS server gave us less data than we asked for when
  we tried to read something.
# short write: The NFS server wrote less data than we asked it to.
# jukebox delay: How many times the NFS server told us _EJUKEBOX_,
  which is theoretically for when the server is slowly retrieving
  something from offline storage. I doubt that you will ever see this
  from normal servers.

# pnfs read: A count of NFS v4.1+ pNFS reads.
# pnfs write: A count of NFS v4.1+ pNFS writes.

All of the VFS operations are for VFS level file and address space
operations. Fully understanding what these counters mean requires
understanding when those operations are used, for what, and why. I don't
have anywhere near this level of understanding of the Linux VFS layer,
so my information here should be taken with some salt.

As you can see from my example _events:_ line, some events are common,
some are rare (eg #22, silly renames, of which there have been 24 over
the lifetime of this NFS mount), and some basically never happen (eg
everything from #23 onwards). Looking at our own collection of quite
a lot of NFS v3 filesystem mounts, the only thing we've seen even a
handful of (on three filesystems) are short writes. I suspect that those
happen when a filesystem runs out of space on the fileserver.

Disclaimer: I'm somewhat fuzzy on what exactly a number of the events
counted here really represent because I haven't traced backwards from
the kernel code that increments the counters to figure out just what
calls it and what it does and so on.

(This is one reason why the lack of good documentation on _mountstats_
is really frustrating. Decoding a lot of this really needs someone who
actively knows the kernel's internals for the best, most trustworthy
results.)