Systemd and waiting until network interfaces or addresses are configured

August 16, 2019

One of the things that systemd is very down on is the idea of running services after 'the network is up', whatever that means; the systemd people have an entire web page on the subject. This is all well and good in theory, but in practice there are plenty of situations where I need to only start certain things after either a named network interface is present or an IP address exists. For a concrete example, you can't set up various pieces of policy based routing for an interface until the interface actually exists. If you're configuring this on boot in a systemd based system (especially one using networkd), you need some way to insure the ordering. Similarly, sometimes you need to listen only on some specific IP addresses and the software you're using doesn't have Linux specific hacks to do that when the IP address doesn't exist yet.

(As a grumpy sysadmin, I actually don't like the behavior of binding to an IP address that doesn't exist, because it means that daemons will start and run even if the system will never have the IP address. I would much rather delay daemon startup until the IP address exists.)

Systemd does not have direct native support for any of this, of course. There's no way to directly say that you depend on an interface or an IP address, and in general the dependency structure has long been under-documented. The closest you can get to waiting until a named network interface exists is to specify an After= and perhaps a Want= or a Requires= on the pseudo-unit for the network interface, 'sys-subsystem-net-devices-<iface>.device'. However, as I found out, the lack of a .device unit doesn't always mean that the interface doesn't exist.

You might think that in order to wait for an IP address to exist, you could specify an After= for the .device unit it's created in and by. However, this has historically had issues for me; under at least some versions of systemd, the .device unit would be created before the IP address was configured. In my particular situation, what worked at the time was to wait for a VLAN interface .device that was on top of the real interface that had the IP address (and yes, I mix tagged VLANs with an untagged network). By the time the VLAN .device existed, the IP address had relatively reliably been set up.

If you're using systemd-networkd and care about network interfaces, the easiest approach is probably to rely on systemd-networkd-wait-online.service; how it works and what it waits for is probably about as good as you can get. For IP addresses, as far as I know there's no native thing that specifically waits until some or all of your static IP addresses are present. Waiting for systemd-networkd-wait-online is probably going to be good enough for most circumstances, but if I needed better I would probably write a shell script (and a .service unit for it) that simply waited until the IP addresses I needed were present.

(I continue to think that it's a real pity that you can't configure networkd .network files to have 'network up' and 'network down' scripts, especially since their stuff for routing and policy based routing is really very verbose.)

PS: One of the unfortunate effects of the under-documented dependency structure and the lack of clarity of what to wait on is a certain amount of what I will call 'superstitious dependencies', things that you've put into your systemd units without fully understanding whether or not you needed them, and why (often also without fully documenting them). This is fine most of the time, but then one day an unnecessary dependency fails to start or perhaps exist and then you're unhappy. That's part of why I would like explicit and reliable ways to do all of this.


Comments on this page:

After reading the systemd web page on running services after the network is up, it makes me even more anti-systemd than I was before. Basically they want to tell every sysadmin how to configure their system, instead of providing the mechanism and tools to support whatever configuration the sysadmin judges will work best for their particular needs.

Some of the systemd requirements just seem daft to me. For example:

"If you write a server: listen on [::], [::1], 0.0.0.0 and 127.0.0.1 only."

So if I want my server to, for example, only listen on the LAN address that is behind my firewall, I'm SOL?

Note also that the web page appears to think this is a "developer" issue--why can't those developers just write software that works? But in fact it's a sysadmin issue, because the developer can't possibly know all the possible network configurations that a sysadmin might want to use and should not presume to tell sysadmins what would be best for them. That's why servers have configuration files to let sysadmins tell them things like "I only want you to listen on these network addresses", so that the sysadmin doesn't have to have exposed ports outside a firewall, for example.

Even Canonical can't get systemd to work reliably. Our cron server recently stopped sending error emails every time it reboots.

Apparently, systemd-resolved comes up, a bunch of network services are launched, it goes down, it comes back, it doesn't resolve DNS for a few more seconds. Worse, all the logs about this don't make it to our central logging platform, because we can't forward them without DNS. In the end, postfix can't send cron's email because the postfix ExecStartPre copied a broken /etc/resolv.conf to the chroot in the moment when systemd-resolved is down.

I ended up adding a path unit to monitor /etc/resolv.conf for changes; the service it starts will then copy that file to Postfix's chroot.

As part of this whole issue, I read the systemd networking page, but it didn't really help. Everything looked correct to me, but broke in practice.

I suspect there are race-conditions or a timing gap when bringing up a VPN connection at boot time. I filed https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/610

Bill Dietrich

By William Manley at 2023-03-23 10:58:37:

You can add:

   ExecStartPre=/lib/systemd/systemd-networkd-wait-online -i eth0 -o routable

to your .service file to wait for a specific interface to be up/in a specific state before the service starts.

Systemd 251 (2022-05-21) also added a template unit `systemd-networkd-wait-online@<interface>.service` allowing you to do the same, but with unit dependencies. The `ExecStartPre=` approach provides more flexibility however as you can specify the `-o` option requiring the interface to be in a specific "operational state".

Still: neither allow you to wait for a specific IP address.

https://www.freedesktop.org/software/systemd/man/systemd-networkd-wait-online.service.html

-- Will

Written on 16 August 2019.
« Getting LSP-based editing working for Go in GNU Emacs
A gotcha with Fedora 30's switch of Grub to BootLoaderSpec based configuration »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri Aug 16 00:26:26 2019
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.