My Fedora office workstation has wound up with a complicated network setup and to go with it, a complicated set of policy based IP routing rules that are intended to create interfaces with dual identity routing and isolation (see also). Right now, my workstation is on two public networks and a shifting number of internal "sandbox" RFC 1918 networks that are behind firewalls, serves as the Wireguard endpoint for my home machine's connection, and now has a NAT'd virtual network for virtual machines. The result of this accreting complexity is that I spent a bunch of today fighting with my policy based routing rules. So now I have some thoughts on keeping things straight here.

For every interface, you (I) need to keep track of what outgoing packets on that interface should be able to reach, including what default route they should have. For everything other than the default interface, this needs to be in a separate routing table because you're going to have to write ip rules to steer outgoing traffic from your IP address on that network in the right way, that will look like:

from <my-net-x-ip> iif lo lookup <net-x-routes>

(You'll need a rule priority.)

If you have additional network routes through any interface, you also need to put them all in a table (without a default route) that will be checked for local traffic that hasn't already picked an IP, very late in the rule list:

# just before the end of rules
from all iif lo look lookup <special-routes>

If your machine will forward any sort of traffic you need to keep track of which networks incoming packets on each interface should be able to reach. The easy way to do this is to set up a routing table for each thing you're a gateway for (in my case the Wireguard tunnel and the new NAT virtual network) and then add ip rule entries to look up in that table for appropriate interfaces. Remember that this includes locally generated packets:

from all iif lo lookup <nat-route>
from all iif lo lookup <wireguard-route>

from all iif <primary-interface> lookup <wireguard-route>

(These routing tables just contain a route to the thing itself, eg '192.168.x.0/24 dev virbr1 src 192.168.x.y'.)

After you've set up what packets from each network should be able to reach, if anything, you need to blackhole all other traffic (with a higher numbered rule):

from all iif <interface> blackhole

(If you don't route anything you don't need to specifically blackhole interfaces, because you won't have turned on IP forwarding and any traffic not for you will get implicitly blackholed.)

Your default IP routes ('ip route list') should contain your primary interface's default route and all your directly attached networks, which will normally be added automatically as they get configured. The default route here will normally get used for traffic you initiate.

For my sins, NAT adds at least three complications to this, because NAT changes IP addresses instead of reinjecting packets. First, the internal network that you're doing NAT for must be steered to the routing table for the network your NAT gateway IP is on:

from all iif virbr1 lookup <net-x-route>

Second, the interface for the network your NAT gateway IP is on must be allowed to reach the NAT network, so that return traffic actually works:

from all iif <net-x-iface> lookup <nat-route>

Finally, I believe that you need a special rule for all locally generated traffic that is before all source IP specific rules, so that you can route to the NAT network:

from all iif lo lookup <nat-route>

The need for this is because packets from the NAT network to any of your local IP addresses will get handled locally, without routing, and so probably won't be NAT'd. This is partly because of the very first 'ip rule' rule that matches incoming traffic against all of your local IPs, '0: from all lookup local', and partly because NAT'ing is usually done in the POSTROUTING iptables chain, which obviously doesn't apply if you're not routing.

I thought I was going to say something about the ever popular subject of rp_filter because it was one of the things I wrestled with today, but after looking at things more I think that I should just set rp_filter to 0 globally and stop worrying about it. I don't think it's protecting me from anything and I know it was periodically causing me problems that were mysterious at the time.

(I would still have had some problems with rp_filter at 0, but at least they would have been more obvious problems instead of packets just vanishing inside the network stack.)

PS: Before I started writing this entry, I thought it would be short and simple. It all seemed so straightforward while it was in my head (just have 'reached through this network' and 'this network can reach' tables and rules entries), but the devil is in the details.

(As a cross reference, see also using policy based routing to isolate a testing interface, which covers an approach when you're only writing rules for some interfaces instead of all of them.)