Our new way of waiting for the network to be "up" in systemd's world

November 9, 2021

Systemd has a long standing philosophical objection to waiting until the network is up; they have an entire web page on the subject. Never the less, we need to do this (like many sysadmins). I've written before about this, and if you're using systemd-networkd either directly or through Ubuntu's netplan, you can in theory use systemd-networkd-wait-online.service. Usually it works, but today we discovered that it didn't on some of our Ubuntu 18.04 servers (the specifics of this issue are beyond the scope of this entry). Since we needed a way to fix the issue, we opted to solve our problem with a hammer.

All of our servers have static IP addresses and are always physically connected to the network unless something has gone terribly wrong. This means that for us, what's needed for the network to come up is for the kernel to probe the hardware for the network ports, then for some combination of Ubuntu's Netplan and systemd-networkd to run and configure the interfaces. We don't have to wait for DHCP or for a cable to be plugged in or any of the other things that can make it complicated to decide when the network is "up"; we simply need to wait for other local software to run and settle.

One general sign that all of this probing and configuring has happened is that at least one of the machine's interfaces has an IP(v4) address. So our brute force solution is a script that waits for this to be true, using a core loop of:

while true; do
   n="$(ip -br -4 addr list | grep -cv '^lo ')"
   if [ "$n" -gt 0 ]; then
   sleep 1

We don't care which interface has an IPv4 address, and we don't care how many do, or how many IPv4 addresses have been configured. Since all we're waiting for is for networkd to have run and set things up, we assume it will do all of this basically at the same time. This saves us having to do any parsing of the machine's Netplan or networkd configuration to see how many interfaces and IPv4 addresses there should be.

(If we thought there might be a little bit of a delay between the first IPv4 address being set and networkd finishing all of its configuration, we could add a sleep of a second or two at the end of the script.)

This script is then wired to a new systemd .service:

After=systemd-networkd.service systemd-networkd-wait-online.service



(The After= settings are there so that the script can spend less time sitting around twiddling its thumbs, and it's more likely that it will succeed on the first check without sleeping at all.)

To be safe, we time out (in the .service unit) after 60 seconds. It shouldn't ever take anywhere near 60 seconds to probe motherboard or PCIe card network hardware and then run networkd (and perhaps Netplan), so if we hit the timeout it's unlikely that waiting longer will change the situation on its own.

(This wouldn't work so neatly for IPv6; we'd have to distinguish between the automatically generated link-local fe80::/10 addresses and global IPv6 addresses that would be set by networkd.)

We could go further to wait for more indications of networking being fully available. Two obvious ones are for an interface with IPv4 addresses to also have carrier and for a default route to be set. But the latter assumes all of our hosts will always have a default route set (which may not always be the case) and the former is harder to test, so for now we've opted not to do either. In practice, networkd sets the default route very shortly after it sets the IP address on the interface, so there is not much of a window for our script to exist and then other .service units to start running without a default route.

(Also, most of our servers are on a single subnet and all of the boot-time services they need to talk to are on that subnet, so missing a default route usually won't stop us from doing things like synchronizing our time through NTP.)

Comments on this page:

From at 2021-11-10 05:16:03:

Yes that script is a bit of a hammer... :-)

The original "hammer" of the UNIX Edition VII and BSD service supervision was that services in each run-level would start in "phases", and each phase (e.g. device setup, filesystem mount, core daemons, networking setup, network daemons, ...) would start only when all processes in the previous phase had been started (and hopefully had the time to initialize themselves). The script here reverts a bit to that logic.

So it goes a bit against the grain of 'systemd' where the goal is that it supervises every task in as fine grained a way as possible so it can start as few processes as late as possible mainly to speed up desktop (but also server) boot.

That is BTW the rationale for 'systemd' to also do package management and network management: to start on demand service "X", that needs network setup, and is contained in package "X-service", 'systemd' has to first dot the task of installing it, only when fully installed 'systemd' has to the task of bringing up the relevant bit of networking, and then it can do the task of starting the service daemon. That is all a consequence of 'systemd' being task oriented rather than event oriented (where events could be produced by subsystems unrelated to 'systemd').

Note: I think the author of 'systemd' was confused by 'upstart' being event based and its default process wrappers being push based; 'upstart' could equally have worked with pull based process wrappers.

Note: Ironically one of the most annoying aspects of 'systemd' is WantedBy which implies a (conditional) "push" type logic, and is very commonly used.

I had a problem a while back when one of my servers came up faster than the switch it was connected to. This meant that IPv6 duplicate address detection times out and prevented the static IPv6 addresses from being configured. These boxes are running sysvinit (because systemd was not ready for production when I set them up) so there was nothing that could add the addresses later when the network came up. So I turned off DAD on the static interfaces.

I suppose this is something to revisit when I modernise these servers, but whatever the mechanism, it needs to be super reliable, because very little works without recursive DNS servers!

From at 2021-11-10 11:39:18:

(This wouldn't work so neatly for IPv6; we'd have to distinguish between the automatically generated link-local fe80::/10 addresses and global IPv6 addresses that would be set by networkd.)

ip addr list scope global will exclude both the link-scope fe80::/10 and the host-scope ::1 &

(It'll also exclude the obsolete site-scope fec0::/10 but chances of your servers using those are pretty slim.)

Written on 09 November 2021.
« Soon to expire TLS certificates aren't necessarily a problem
People will always exploit presentation, because presentation matters »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Tue Nov 9 23:15:55 2021
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.