== Processes waiting for NFS IO do show in Linux _%iowait_ statistics

Suppose that you have a machine that does decent amounts of both
local disk IO and NFS IO and it's not performing as well as you'd
like. Tools like _vmstat_ show that it's spending a significant
amount of time in [[_%iowait_ LinuxIowait]] while [[your (local)
disk stats tool MxiostatPointer]] is somewhat ambivalent but suggests
that the local disks are often not saturated. Can you safely conclude
that your system is spending a bunch of its time waiting on NFS IO
and this is what the _%iowait_ numbers are reflecting?

As far as I can tell from both experimentation and some reading of
the kernel source, the answer is yes. Waiting for NFS IO shows up
in _%iowait_. A NFS client with otherwise inexplicable _%iowait_
times is thus waiting on NFS IO because your fileservers aren't
responding as fast as it would like.

(Crudely simplifying, from the kernel source it very much looks
like the same mechanisms drive _%iowait_ as drive things like
vmstat's _b_ column ('busy' processes, processes waiting in
uninterruptible sleep) and in fact the Linux load average itself,
and processes waiting on NFS IO definitely show up in the latter
two.)

You might wonder why I bothered asking such an obvious question.
The simple answer is that Unix systems have had a historical habit
of not considering remote filesystem IO to be 'real' disk IO. In
the old days it would have been perfectly in character for _%iowait_
to only reflect IO to real disks. Current manpages for _vmstat_,
_top_, and so on do describe _%iowait_ generally as eg 'time spent
waiting for IO' (without restricting it to disks) but [[my old
DiskIOStats]] [[habits DiskIOStatsII]] die hard.

=== Sidebar: NFS client performance information

It turns out that while I wasn't looking Linux has gained quite
detailed NFS client performance statistics in the form of a whole
barrage of (per-filesystem) stuff that's reported through
_/proc/self/mountstats_. Unfortunately both documentation on what's
in _mountstats_ and good tools for monitoring it seem to be a bit
lacking, but a lot of information is there if you can dig it out.

(See [[_nfsiostats_
http://sebastien.godard.pagesperso-orange.fr/man_nfsiostat.html]]
from the [[sysstat package
http://sebastien.godard.pagesperso-orange.fr/index.html]] for one
thing that reads it. Note that this can be compiled on, say, Ubuntu
10.04 even though 10.04 didn't package it.)