2009-09-03
An interesting issue with doing NFS over TCP (apparently)
We have a lot of NFS filesystems and, like most people today, we use NFS over TCP instead of over UDP. But this leads to a problem; sometimes when our systems reboot, they can't mount all of the NFS filesystems on the first attempt. It generally takes several minutes to get to a state where all of them are mounted.
(We don't use the automounter, so we mount everything at boot; we have our own solution for the problems the automounter is trying to solve.)
The cause turns out to be interesting; we're running out of reserved
ports, apparently ultimately because of
all of the NFS mount requests we make in close
succession. Like the NFS server, the NFS mount daemon usually requires you to talk to it from a reserved
port, and although each conversation between mount
and mountd
is
short-lived and we only make one mount request at a time, Linux can wind
up not letting you reuse the same source port to talk to the same mount
daemon for a timeout interval. It turns out that we have enough NFS
mounts from few enough fileservers that we can temporarily run out of
reserved ports that mount
can use to talk to a particular fileserver's
mountd
.
(This is the TIME_WAIT
timeout for a given combination of source IP
address, source port, destination IP address, and destination port. The
destination port on a given fileserver is always fixed, so effectively
the only variable is the source port, and there's a limited supply of
reserved ports that mount
is willing to use.)
Our experience is that this doesn't happen when we use NFS over UDP (we
have one system that does this, for reasons that may not be applicable
any more). Having written this entry, I'm now not sure why this is
so, since although the actual NFS traffic is UDP-based, mount
is
presumably still talking to mountd
with TCP and so is still using up
reserved ports there.
(This is somewhat related to an earlier problem with NFS mounts that we've had.)
Programming blindness and security
Here is a thought and a theory that has recently struck me.
One reason that writing secure programs is hard is because programmers have a kind of blindness about our own code. Instead of seeing the code as it really is, we tend to see the code as we imagine it in our heads, and as a result we see what we wrote it to do, not what it actually does.
(This is not usually a literal blindness, although we can do that too, but a disconnect between what the the text says and what we 'know' that it does.)
In ordinary code, this just makes your bugs harder to find (and leaves
you wondering how you could have possibly missed such an obvious mistake
once you find it). In security sensitive code it leads to holes that
you can't see because of course you didn't intend to create holes. If
you wrote code to introspect operations on the fly by loading a file
from a directory, you don't see that it could load a file from anywhere
with judicious use of ../
in the filename, because it's not what
you wrote the code to do. Of course the code doesn't let you load
arbitrary files because you didn't write the code to load arbitrary
files, you wrote it to load files from a directory.
Effective debugging in general requires being able to escape this blindness, but at least you have an initial shove in that you know that there's a bug in your code. Checking for security holes is even harder because there is nothing obviously wrong, nothing to kick you out of your mindset of blindness and get you to take a second look.
This leads to obvious but interesting (to me) thoughts about the effectiveness of things like pair programming, code reviews, and security audits. From the angle of this theory, these work in part because they expose your code to people who have less of a chance of being caught up in the blindness.
(I suspect that this is not an original thought.)