Wandering Thoughts archives

2009-09-03

An interesting issue with doing NFS over TCP (apparently)

We have a lot of NFS filesystems and, like most people today, we use NFS over TCP instead of over UDP. But this leads to a problem; sometimes when our systems reboot, they can't mount all of the NFS filesystems on the first attempt. It generally takes several minutes to get to a state where all of them are mounted.

(We don't use the automounter, so we mount everything at boot; we have our own solution for the problems the automounter is trying to solve.)

The cause turns out to be interesting; we're running out of reserved ports, apparently ultimately because of all of the NFS mount requests we make in close succession. Like the NFS server, the NFS mount daemon usually requires you to talk to it from a reserved port, and although each conversation between mount and mountd is short-lived and we only make one mount request at a time, Linux can wind up not letting you reuse the same source port to talk to the same mount daemon for a timeout interval. It turns out that we have enough NFS mounts from few enough fileservers that we can temporarily run out of reserved ports that mount can use to talk to a particular fileserver's mountd.

(This is the TIME_WAIT timeout for a given combination of source IP address, source port, destination IP address, and destination port. The destination port on a given fileserver is always fixed, so effectively the only variable is the source port, and there's a limited supply of reserved ports that mount is willing to use.)

Our experience is that this doesn't happen when we use NFS over UDP (we have one system that does this, for reasons that may not be applicable any more). Having written this entry, I'm now not sure why this is so, since although the actual NFS traffic is UDP-based, mount is presumably still talking to mountd with TCP and so is still using up reserved ports there.

(This is somewhat related to an earlier problem with NFS mounts that we've had.)

linux/TCPNFSMountProblem written at 22:49:01;

Programming blindness and security

Here is a thought and a theory that has recently struck me.

One reason that writing secure programs is hard is because programmers have a kind of blindness about our own code. Instead of seeing the code as it really is, we tend to see the code as we imagine it in our heads, and as a result we see what we wrote it to do, not what it actually does.

(This is not usually a literal blindness, although we can do that too, but a disconnect between what the the text says and what we 'know' that it does.)

In ordinary code, this just makes your bugs harder to find (and leaves you wondering how you could have possibly missed such an obvious mistake once you find it). In security sensitive code it leads to holes that you can't see because of course you didn't intend to create holes. If you wrote code to introspect operations on the fly by loading a file from a directory, you don't see that it could load a file from anywhere with judicious use of ../ in the filename, because it's not what you wrote the code to do. Of course the code doesn't let you load arbitrary files because you didn't write the code to load arbitrary files, you wrote it to load files from a directory.

Effective debugging in general requires being able to escape this blindness, but at least you have an initial shove in that you know that there's a bug in your code. Checking for security holes is even harder because there is nothing obviously wrong, nothing to kick you out of your mindset of blindness and get you to take a second look.

This leads to obvious but interesting (to me) thoughts about the effectiveness of things like pair programming, code reviews, and security audits. From the angle of this theory, these work in part because they expose your code to people who have less of a chance of being caught up in the blindness.

(I suspect that this is not an original thought.)

programming/ProgrammingBlindness written at 00:13:07;


Page tools: See As Normal.
Search:
Login: Password:

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.