Using policy based routing to isolate a testing interface on Linux
The other day I needed to do some network bandwidth tests to and from one of our sandbox networks and wound up wanting to use a spare second network port on an already-installed test server that was fully set up on our main network. This calls for policy based routing to force our test traffic to flow only over the sandbox network, so we avoid various sorts of asymmetric routing situations (eg). I've used Linux's policy based routing and written about it here before, but surprisingly not in this specific situation; it's all been in different and more complicated ones.
So here is what I need for a simple isolated testing interface, with commentary so that when I need this again I don't have just the commands, I also can re-learn what they're doing and why I need them.
- First we need to bring up the interface itself. For quick
testing I just use raw
ip link set eno2 up ip addr add dev eno2 172.21.1.200/16
- We need a routing table for this interface's routes and a
routing policy rule that forces use of them for traffic to
and from our IP address on
ip route add 172.21.0.0/16 dev eno2 table 22 ip route add default via 172.21.254.254 table 22 ip rule add from 172.21.1.200 iif lo table 22 priority 6001
We need the local network route for good reasons. The choice of table number is arbitrary.
By itself this is good enough for most testing. Other hosts can
connect to your 172.21.1.200 IP and that traffic will always flow
eno2, as will outgoing connections that you specifically
bind to the 172.21.1.200 IP address using things like
argument or Netcat's
-s argument. You can also talk directly to
things on 172.21/16 without having to explicitly bind to 172.21.1.200
first (ie you can do '
ping 172.21.254.254' instead of needing
ping -I 172.21.1.200 172.21.254.254').
However, there is one situation where traffic will flow over the
wrong network, which is if another host in 172.21/16 attempts to
talk to your public IP (or if you try to talk to 172.21/16 while
specifically using your public IP). Their outbound traffic will
come in on
eno1, but because your machine knows that it can talk
to them directly on
eno2 it will just send its return traffic
that way (probably with odd ARP requests).
What we want is to use the direct connection to 172.21/16 in only
two cases. First, when the source IP is set to 172.21.1.200 in some
way; this is already covered. Second, when we're generating outgoing
traffic locally and we have not explicitly picked a source IP; this
allows us to do just '
ping 172.21.254.254' and have it flow over
eno2 the way we expect. There are a number of ways we could do
this, but it turns out that the simplest way goes as follows.
- Remove the global routing table entry for
ip route del 172.21.0.0/16 dev eno2
(This route in the normal routing table was added automatically when we configured our address on
- Add a new routing table with the local network route to 172.21/16
and use it for outgoing packets that have no source IP assigned
ip route add 172.21.0.0/16 dev eno2 src 172.21.1.200 table 23
ip rule add from 0.0.0.0 iif lo lookup 23 priority 6000
The nominal IP address
INADDR_ANYis what the socket API uses for 'I haven't set a source IP', and so it's both convenient and sensible that the kernel reuses it during routing as 'no source IP assigned yet' and lets us match on it in our rules.
(Since our two rules here should be non-conflicting, we theoretically could use the same priority number. I'm not sure I fully trust that in this situation, though.)
You can configure up any number of isolated testing interfaces following this procedure. Every isolated interface needs its own separate table of its own routes, but table 23 and its direct local routes are shared between all of them.