An interesting issue with doing NFS over TCP (apparently)

We have a lot of NFS filesystems and, like most people today, we use NFS over TCP instead of over UDP. But this leads to a problem; sometimes when our systems reboot, they can't mount all of the NFS filesystems on the first attempt. It generally takes several minutes to get to a state where all of them are mounted.

(We don't use the automounter, so we mount everything at boot; we have our own solution for the problems the automounter is trying to solve.)

The cause turns out to be interesting; we're running out of reserved ports, apparently ultimately because of all of the NFS mount requests we make in close succession. Like the NFS server, the NFS mount daemon usually requires you to talk to it from a reserved port, and although each conversation between mount and mountd is short-lived and we only make one mount request at a time, Linux can wind up not letting you reuse the same source port to talk to the same mount daemon for a timeout interval. It turns out that we have enough NFS mounts from few enough fileservers that we can temporarily run out of reserved ports that mount can use to talk to a particular fileserver's mountd.

(This is the TIME_WAIT timeout for a given combination of source IP address, source port, destination IP address, and destination port. The destination port on a given fileserver is always fixed, so effectively the only variable is the source port, and there's a limited supply of reserved ports that mount is willing to use.)

Our experience is that this doesn't happen when we use NFS over UDP (we have one system that does this, for reasons that may not be applicable any more). Having written this entry, I'm now not sure why this is so, since although the actual NFS traffic is UDP-based, mount is presumably still talking to mountd with TCP and so is still using up reserved ports there.

(This is somewhat related to an earlier problem with NFS mounts that we've had.)