Combining dual identity routing and isolated interfaces

In theory, combining dual identity routing with isolated interfaces should be simple. However, it turns out that there is a small trick to it.

This came up today because I am in the process of migrating my office machine from one network to another. This means that I currently have three active interfaces on the machine (old IP address, new IP address, and an internal network), plus one end of a GRE tunnel, and as part of the migration I wanted to force the routing to work right, just in case.

The problem with the straightforward combination is that it will try to force locally generated packets from one of the machine's IP addresses out the appropriate network interface even if it is talking to the other end of the GRE tunnel. This is unlikely to work very well; at the best it will bounce off the gateway router and come right back.

The solution is to steal packets for the GRE tunnel before they get handled by the source IP address rules. Usefully, the isolated interfaces part of this has already defined a routing table with exactly the right IP addresses in it. So we write the ip rule statement as:

# table 10 is from the isolated interfaces
ip rule add iif lo priority 4999 table 10

ip route add default via GW1 dev eth0 table 20
ip rule add from IP1 iif lo priority 5000 table 20

ip route add default via GW2 dev eth1 table 21
ip rule add from IP2 iif lo priority 5001 table 21

ip route add default via GW3 dev eth2 table 22
ip rule add from IP3 iif lo priority 5002 table 22

As a useful side effect, this makes it possible for me to fully test the firewall rules protecting the internal network, since now my machine is willing to actually route packets through the firewall.

(I am tempted to package the ip route/ip rule bits up as a script, just so I can conveniently bring up scratch IP addresses on various VLANs for exactly this sort of firewall testing. The only tricky bit is automatically picking the table number, as ip rule seems happy to have a bunch of rules with the same priority.)

Alas, I am now left with a mystery: according to the policy routing rules, it looks like a packet from IP1 to an address on that subnet should get routed via the gateway (and similarly for the other networks), as the main routing table that has the local-network routing information is only looked at by a rule with priority 32766, much lower than the priority 5000 rule for locally generated packets with source addresses of IP1.

(I just checked and the explicit rule to steal GRE tunnel traffic is necessary; without it the packets get shoved out on eth0 instead.)

Important update, November 20th: it turns out that I was wrong and packets from IP1 to the subnet were getting routed via the gateway. See DualIdentityIsolationII for the fix.