Wishing for a simple way to set up multi-interface symmetric routing on Linux

June 27, 2022

For neither the first nor the last time, I've wound up in a situation where it would be quite useful for one of our machines to have what I will describe as simple symmetric routing across multiple interfaces. What I mean by this is a situation where each of the host's IP addresses is associated with an interface and when packets go out with a particular IP address, they use that interface (and the interface's default route). I call this "symmetric routing" because it makes the inbound and outbound paths the same for a given connection, which is not the case by default for a host with multiple interfaces today.

Setting this up with Linux's policy based routing is straightforward and almost mechanical. However, the setup has a lot of moving parts and there's no current automation for it that I know of. You can build your own, of course, but then that means you're stuck maintaining and operating your own automation; at that point you (we) start asking if you (we) really need symmetric routing, or if it's just a nice to have thing.

If you're directly using systemd-networkd, you can probably build something out of [Route] sections and [RoutingPolicyRule] sections, but keeping all of the sections organized for each interface and keeping the table numbers straight is up to you. Ubuntu's netplan can express similar things in its routing and routing-policy sections, but once again you're left to hand-craft everything to keep it organized (a look at the netplan examples may help get the syntax and placement of directives right). However, I'm not convinced that netplan can be made to work correctly for this because I don't see how to add direct subnet routes to tables in netplan, and direct subnet routes are required in some situations.

(It's also not always clear that you've considered all of the corner cases, especially if you're trying for a simple setup. As I've found, there can be quite a number of corner cases, some of which aren't obvious and usually appear to work.)

I don't expect the Linux kernel to have a simple configuration option or way to do this. The Linux kernel traditionally provides a bunch of low level networking options and it's up to you to build what you want out of them. But I would like things like systemd's networkd and Ubuntu's netplan to have a simple way of configuring something like this, one that reduces the amount of make-work and insures that you've covered all of the corner cases.

(I would be surprised to get it, though. It's a little bit amazing that we have policy based routing support in systemd's networkd and Ubuntu netplan.)

PS: I've historically done this in two different ways, one as isolated interfaces for testing purposes and the other as my general isolated networks on my desktop. I'm not sure which approach works better, and that sort of illustrates why I'd like to have this all handled by networkd or netplan.

Comments on this page:

By Isaac at 2022-06-28 12:53:11:

Hello, Have you considered using VRFs ? You might need additional packages such as those implemented by Vyatta

By cks at 2022-06-28 14:08:57:

I've been reading about VRFs, but as far as I can tell from the reading (and from some experimentation), VRFs create too much isolation in that programs can't use a VRF without special measures. Symmetric routing is more convenient because you can reach all directly connected networks normally (by just using an IP address on them) and you only need to take special measures if you want to use a specific interface for traffic you initiate.

By Isaac at 2022-06-28 16:43:44:

My understanding is that you have a physical server with multiple physical interfaces, also with the possibility of multiple addresses per interface.

There could be multiple virtual machines / containers located on this physical server. My proposal from this viewpoint, is to introduce a router as another process/VM on the same host.

Each logically addressable component (VM or container), will have the option of one or more logical interfaces connecting to the virtual router - and based on the logical network/subnet will be assigned to the appropriate VRF.

The complexity will be moved to the routing instance. Of course you could also buy a router from Cisco/Juniper/etc and create virtual LANS from the router to the host, then apply VRFs on the hardware appliance.

The key issue I believe this will solve is configuration consistency, you will not have to worry about tweaking the VM/container level routing tables. I have seen strange behavior when configuring route policies/multiple routing tables on linux VMs. But I may be off the mark.

By cks at 2022-06-28 17:07:03:

We don't have containers or VMs on the server in question. It just has multiple interfaces, each of them with an IP address. To avoid assorted anomalies, we would like routing to be symmetric: if an outside packet came in on a given interface, the reply to it should go out the same interface. We'd also like to be able to easily force the use of a specific interface and a specific set of routes for outbound connections and other traffic originating on the machine.

(Symmetric routing is vital if different interfaces are behind different firewalls, and it's important if you think some network paths may fail while others continue working. If you received a packet over a given interface and its path, that interface and path has the highest chance of a reply packet getting to the origin.)

VRFs don't do this simply because in various circumstances they don't automatically select the right outgoing interface. Plain 'ping' is one case, and if I'm understanding things right, traffic from a UDP socket not bound to a specific IP may be another one.

You should be able to add link routes in netplan 0.104. See this and the pull request linked from it: https://bugs.launchpad.net/netplan/+bug/1805038

Written on 27 June 2022.
« Modern disk sizes and powers of two
What symmetric and asymmetric IP routing are »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Jun 27 22:50:31 2022
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.