Chris's Wiki :: blog/linux/NetplanRoutesProblem

Our problem with Netplan and routes on Ubuntu 18.04

March 7, 2019

Today I've wound up getting back to our netplan dysfunction, so I think it's time to write a blog entry. Spoiler: highly specific network device names and configurations that can only be attached to or specified for a named network device interact very badly at scale.

We have a bunch of internal 'sandbox' networks, which connect to our main server subnet through various routing firewalls. Obviously the core router on our server subnet knows how to route to every sandbox; however, we also like to have the actual servers to have specific sandbox subnet routes to the appropriate routing firewall. For some servers this is merely vaguely nice; for high traffic servers (such as some NFS fileservers) it may be pretty important to avoid a pointless traffic bottleneck on the router. So many years ago we built a too-smart system to automatically generate the appropriate routes for any given host from a central set of information about what subnets were behind which gateway, and we ran it on boot to set things up. The result is a bunch of routing commands:

ip route add 10.63.0.0/16 via 128.100.3.5
ip route add 10.70.0.0/16 via 128.100.3.4
ip route add 172.31.0.0/16 via 128.100.3.6
[...]

This system is completely indifferent to what the local system's network interface is called, which is good because in our environment there is a huge assortment of interface names. We have eno1, enp3s0f0, enp4s0f0, enp4s0, enp11s0f0, enp7s0, enp1s0f0, and on and on.

All of this worked great for the better part of a decade, until Ubuntu 18.04 came along with netplan. Netplan has two things that together combine to be quietly nearly fatal to what we want to do. First, the netplan setup on Ubuntu 18.04 will wipe out any 'foreign' routes it finds if and when it is re-run, which happens every so often during things like package upgrades. Second, the 18.04 version of netplan has no way to specify routes that are attached to a subnet instead of a specific named interface. If you want netplan to add extra routes to an interface, you cannot say 'associate the routes with whatever interface is on subnet <X>'; instead, you must associate the routes with an interface called <Y>, for whatever specific <Y> is in use on this system. As mentioned, <Y> is what you could call highly variable across our systems.

(Netplan claims to have some support for wildcards, but I couldn't get it to work and I don't think it ever would because it is wildcarding network interface names alone. Many of our machines have more than one network interface, and obviously only one of them is on the relevant subnet (and most of the others aren't connected to anything).)

The result is that there appears to be no good way for our perfectly sensible desire for generic routing to interact well with netplan. In a netplan world it appears that we should be writing and re-writing a /etc/netplan/02-cslab-routes.yaml file, but that file has to have the name of the system's current network interface burned into it instead of being generic. We do shuffle network interfaces around every so often (for instance to move a system from 1G to 10G-T), which would require us remembering that there is an additional magic step to regenerate this file.

There are various additional problems here too, of course. First, there appears to be no way to get netplan to redo just your routes without touching anything else about interfaces, and we very much want that. Second, on most systems we establish these additional sandbox routes only after basic networking has come up and we've NFS mounted our central administrative filesystem that has the data file on it, which is far too late for normal netplan. I guess we'd have to rewrite this file and then run 'netplan apply'.

(Ubuntu may love netplan a whole lot but I certainly hope no one else does.)

Comments on this page:

By Arnaud Gomes at 2019-03-07 02:36:39:

Is there a reason you do not simply deploy some kind of dynamic routing at least on your firewalls and critical servers? Either OSPF or BGP with route reflectors, depending on the exact degree of control you need.

Every time I started to design the kind of setup you use, I ended up just firing a few OSPF daemons and letting them do the job.
   -- A

By theamk at 2019-03-07 10:07:25:

We are planning ubuntu 18.04 upgrade on our systems, and I am planning to skip netplan altogether. Being ubuntu-specific, I am afraid it might go the way of upstart sometimes soon.

Instead, my current plan is to go with systemd-networkd directly. It sounds more annoying, as there are many small files. But at least there is an official way to autogenerate them -- the systemd's generator mechanism -- which might come up handy if I find syntax to be too limiting.

By cks at 2019-03-07 12:48:39:

On dynamic routing: we don't have any experience running OSPF/BGP/etc, so it would be a significant increase in complexity for very little gain for us. We don't have redundant paths and in practice changes in routing are on the level of a few times a year (when we add a new internal subnet or sometimes move one between firewalls).

On (not) using netplan: I endorse this idea today, but we tend to go with Ubuntu's defaults rather than fighting them and at the time we built our 18.04 setup we didn't know any better. For that matter I don't know if it's particularly easy to install an 18.04 machine without netplan through a relatively standard install, or if you need to install with netplan and then manually rip it out afterward and transform the system setup to a networkd-based one.

(I'm pretty sure we could do what we want with networkd alone.)

By nolan at 2019-03-07 18:06:47:

18.04 does have ifupdown2 in the repos, which is far less insane than netplan.

By Trent Lloyd at 2019-03-08 04:27:41:

If you change the name of each "ethernets" sub-section to be a generic name (e.g. "mainif") in advance, then you can then update the routes configuration file without knowing the real interface name.

Inside the "mainif" section in your main configuration file, you would have a "match" section with either name or macaddress.

Then a second yaml file can simply have a 'mainif' section and then your routes and not need to know the name/mac from the first file

Example: https://paste.ubuntu.com/p/Bjf6B38fr4/

networkd's design is showing through netplan here. I was originally going to suggest your own *.network file with only the static route but networkd only recognizes 1 .network file per interface, overrides have to go into a directory with the same name as the original .network file.

The .network file is named by netplan from the same place, so by "default" (as most people or the installer writes it) - it has the interface name (e.g. 'ens3') and now you have to know that name to write a /etc/systemd/network/10-netplan-ens3.network.d/routes.conf file containing the route. This obviously doesn't help since the entire point was to avoid having to know the interface name.

Once we rename this section to a generic name, e.g. 'mainif' we could write /etc/systemd/network/10-netplan-mainif.network.d/routes.conf file - but now netplan will also let us directly write netplan yaml in the same way only referencing the 'mainif' name.

This behavior also explains the trouble you had with the wildcard matching. Netplan supports wildcards but again it's showing through the underlying behavior of networkd. Those wildcards are given to networkd, it says to networkd 'match' this MAC or Name (literally with a networkd [Match] section). If you write a netplan YAML that has routes in a separate file, with a separate ethernet section name such as "my-routes", it will write that into a networkd config file called "my-routes.network" (separate to your ens3.network file). As shown before networkd only recognises a single .network file per interface, so, this configuration though existing and technically sounding correct, won't be applied. The reason it works in the netplan case, is because we are not doing any matching, we are telling it this route belongs to the "mainif" configuration. Netplan actually merges this section onto the end of the /run/systemd/network/10-netplan-mainif.network file but in essence it is exactly the same as writing it separately into the /etc/systemd/network/10-netplan-mainif.network.d/routes.conf as an override file. In either case we needed to know the 'name' of this particular network we are configuring (regardless of the actual interface name, it may or may not be the same).

I have seen similar questions from people before, for example, how to programatically modify the nameservers written into the Netplan YAML. You can't really do this with a 'simple' sed-style script in a reliable fashion, ultimately you need a YAML parser and a bit of logic to do it in a fool proof way. But that logic still needs to know which interface to pick. In a single-interface world you can just pick the only interface, but if you have more than 1 interface then something, somewhere, has to have known how to pick which interface - we can't avoid that. We can do that in advance at setup time by giving the interface a "friendly" name such as "mainif" like I suggested here or otherwise whatever is modifying that file needs to know the actual name (or otherwise pick an interface with some kind of logic, perhaps which interface has the default route or who knows what).

The final question you may ask then, is can we offload this specific behavior to netplan/networkd so that we can have a separate list of routes installed when the relevant interface comes up. You could in theory implement some additional kind of "match" which asks netplan or networkd to match the interface connecting us to the given gateway IP for the route. I would suggest while that might be useful in an ideal network configuration system, in reality if you label your network configs with a functional 'name' rather than based on their interface name as I suggested, you could get away with that level of matching for most tasks. Perhaps we should look at changing the default configurations to show a functional 'name' so that this kind of task is more obvious to the average user?

(For similar reasons I used to rename interfaces on some servers I managed to 'live0', 'nfs0' etc so that we could easily reference the interface name/configuration on servers that might have only a subset of them. In that case we actually renamed the interface with udev. You can actually do that with netplan by setting the 'set-name: <name>' key. With netplan/networkd though for configuration purposes we can still have such a name without actually changing the interface name. But it would still help you when you're running manual commands etc if you did rename it I guess :)

Hope that helps...

Trent @lathiat Lloyd https://twitter.com/lathiat

Written on 07 March 2019.

«	Using Prometheus subqueries to look for spikes in rates

Exploring the mild oddity that Unix pipes are buffered