Suppose that you have a machine that does decent amounts of both local disk IO and NFS IO and it's not performing as well as you'd like. Tools like vmstat show that it's spending a significant amount of time in %iowait while your (local) disk stats tool is somewhat ambivalent but suggests that the local disks are often not saturated. Can you safely conclude that your system is spending a bunch of its time waiting on NFS IO and this is what the %iowait numbers are reflecting?

As far as I can tell from both experimentation and some reading of the kernel source, the answer is yes. Waiting for NFS IO shows up in %iowait. A NFS client with otherwise inexplicable %iowait times is thus waiting on NFS IO because your fileservers aren't responding as fast as it would like.

(Crudely simplifying, from the kernel source it very much looks like the same mechanisms drive %iowait as drive things like vmstat's b column ('busy' processes, processes waiting in uninterruptible sleep) and in fact the Linux load average itself, and processes waiting on NFS IO definitely show up in the latter two.)

You might wonder why I bothered asking such an obvious question. The simple answer is that Unix systems have had a historical habit of not considering remote filesystem IO to be 'real' disk IO. In the old days it would have been perfectly in character for %iowait to only reflect IO to real disks. Current manpages for vmstat, top, and so on do describe %iowait generally as eg 'time spent waiting for IO' (without restricting it to disks) but my old habits die hard.

Sidebar: NFS client performance information

It turns out that while I wasn't looking Linux has gained quite detailed NFS client performance statistics in the form of a whole barrage of (per-filesystem) stuff that's reported through /proc/self/mountstats. Unfortunately both documentation on what's in mountstats and good tools for monitoring it seem to be a bit lacking, but a lot of information is there if you can dig it out.

(See nfsiostats from the sysstat package for one thing that reads it. Note that this can be compiled on, say, Ubuntu 10.04 even though 10.04 didn't package it.)