2024-06-26
The xinetd restart problem with binding ports that we run into
Recently, I said something on the Fediverse:
It has been '0' days since I wished xinetd had an option to exit with a failed status if it couldn't bind some configured ports on startup. (Yes, yes, I know, replace it with systemd socket listeners or something.)
The job of xinetd is to listen on some number of TCP or UDP ports for you, and run things when people connect to those ports. This has traditionally been used to avoid having N different inactive daemons each listening to its own ports, and also so that people don't have to write those daemons at all; they can write something that gets started with a network connection handed to it and talks over that connection, which is generally simpler (you can even use shell scripts). At work, our primary use for xinetd is invoking Amanda daemons on all of the backup clients.
Every so often, Ubuntu releases a package update that wants to restart xinetd (for whatever reason, most obviously updating xinetd itself). When xinetd restarts, the old xinetd process stops listening on its ports and the new xinetd attempts to start listening on what you've configured. Unfortunately this is where a quirk of the BSD sockets API comes in. If there is an existing connection on some port, the new xinetd is unable to start listening on that port again. So, for example, if the Amanda client is running at the time that xinetd restarts, xinetd will not be able to start listening on the Amanda TCP port.
(See the description of SO_REUSEADDR in socket(7). The error
you'll get is 'address already in use', EADDRINUSE. At one
point I thought this was an issue only for
UDP listening, but today it is clearly also a problem for TCP
services.)
When this happens, the current version of xinetd will log messages but happily start up, which means that to general status monitoring it looks like a healthy service. This can make it hard to detect until, for example, some Amanda backups fail because xinetd on the machines to be backed up isn't actually listening for new Amanda connections. This happens to us every so often, which is why I wish xinetd had an option to fail in this situation (and then we'd get an alert, or we could have the systemd service unit set to auto-restart xinetd after a delay).
(Systemd socket units don't so much solve this as work around it by never closing and re-opening the listening socket as part of service changes or restarts.)