2007-09-21
An interesting bind(2) failure
I fired up a version of Dovecot on a testing server today, only to be greeted with:
Fatal: listen(993) failed: Address already in use
That was kind of peculiar, since nothing else was running on the
machine, certainly nothing that should be using the imap-over-SSL port.
I tried starting Dovecot again and got the same error, looked at the
xinetd configuration just in case, tried lsof -i and saw that no
strange daemon was listening, tried connecting to the port and got a
connection refused, and finally wound up straceing the Dovecot process
just in case I had somehow asked it to bind to port 993 twice and it was
the second time around that was failing. Nothing had any enlightenment.
(At this point, as you might imagine, I was both frustrated and worried. Binding to sockets like this is just not supposed to fail mysteriously. I couldn't even suspect SELinux, since this was an Ubuntu machine.)
Finally I ran netstat --inet -a. In the listing of connected ports I
saw a TCP connection between port 993 on the local machine and port 2049
on one of our NFS servers, and the proverbial penny dropped.
What had happened is that the NFS client code uses so-called 'reserved ports' (ports under 1024) locally, starting from 1023 and counting down. Linux won't let you bind a listening port to a port that is already in use for the local end of a connection, and by coincidence we had enough NFS mounts (set up in the right sequence) so that port 993 was in use by the time I tried to start the version of Dovecot that we wanted to test.
The lesson I take away from this is that we should be sure all of our network daemons are started by the time we do NFS mounts, or we may run into this for real someday. And we're lucky that almost all of our mounts are UDP-based, or we would have run into this before now, since we have several hundred NFS mounts and, contrary to what I wrote the other day, it appears that the NFS client creates a new TCP socket for each separate mount.
Sidebar: what reserved ports are and why NFS uses them
On Unix based operating systems, only root is allowed to use local ports of 1023 or below; these are called reserved ports, since they are reserved for root.
As a weak security measure to prevent users on a client machine from forging NFS requests and reading the replies, NFS servers often require that clients talk to them using a reserved port. This way the server has some assurance that it is getting requests from root on the client, not a random user.