Using policy based routing to isolate a testing interface on Linux

July 31, 2017

The other day I needed to do some network bandwidth tests to and from one of our sandbox networks and wound up wanting to use a spare second network port on an already-installed test server that was fully set up on our main network. This calls for policy based routing to force our test traffic to flow only over the sandbox network, so we avoid various sorts of asymmetric routing situations (eg). I've used Linux's policy based routing and written about it here before, but surprisingly not in this specific situation; it's all been in different and more complicated ones.

So here is what I need for a simple isolated testing interface, with commentary so that when I need this again I don't have just the commands, I also can re-learn what they're doing and why I need them.

  • First we need to bring up the interface itself. For quick testing I just use raw ip commands:

    ip link set eno2 up
    ip addr add dev eno2

  • We need a routing table for this interface's routes and a routing policy rule that forces use of them for traffic to and from our IP address on eno2.

    ip route add dev eno2 table 22
    ip route add default via table 22
    ip rule add from iif lo table 22 priority 6001

    We need the local network route for good reasons. The choice of table number is arbitrary.

By itself this is good enough for most testing. Other hosts can connect to your IP and that traffic will always flow over eno2, as will outgoing connections that you specifically bind to the IP address using things like ping's -I argument or Netcat's -s argument. You can also talk directly to things on 172.21/16 without having to explicitly bind to first (ie you can do 'ping' instead of needing 'ping -I').

However, there is one situation where traffic will flow over the wrong network, which is if another host in 172.21/16 attempts to talk to your public IP (or if you try to talk to 172.21/16 while specifically using your public IP). Their outbound traffic will come in on eno1, but because your machine knows that it can talk to them directly on eno2 it will just send its return traffic that way (probably with odd ARP requests). What we want is to use the direct connection to 172.21/16 in only two cases. First, when the source IP is set to in some way; this is already covered. Second, when we're generating outgoing traffic locally and we have not explicitly picked a source IP; this allows us to do just 'ping' and have it flow over eno2 the way we expect. There are a number of ways we could do this, but it turns out that the simplest way goes as follows.

  • Remove the global routing table entry for eno2:

    ip route del dev eno2

    (This route in the normal routing table was added automatically when we configured our address on eno2.)

  • Add a new routing table with the local network route to 172.21/16 and use it for outgoing packets that have no source IP assigned yet:

    ip route add dev eno2 src table 23
    ip rule add from iif lo lookup 23 priority 6000

    The nominal IP address is INADDR_ANY (cf). INADDR_ANY is what the socket API uses for 'I haven't set a source IP', and so it's both convenient and sensible that the kernel reuses it during routing as 'no source IP assigned yet' and lets us match on it in our rules.

(Since our two rules here should be non-conflicting, we theoretically could use the same priority number. I'm not sure I fully trust that in this situation, though.)

You can configure up any number of isolated testing interfaces following this procedure. Every isolated interface needs its own separate table of its own routes, but table 23 and its direct local routes are shared between all of them.

Written on 31 July 2017.
« Link: How does "the" X11 clipboard work?
Why I'll never pick the 'sign in with a Facebook or Google account' option »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Jul 31 22:59:19 2017
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.