A while back I discovered a limitation in Linux's policy based routing, where you couldn't use the straightforward means to flexibly route outgoing traffic on a dual identity machine over different interfaces (for example, to force all SSH traffic to flow over one link, regardless of the destination). At the time I wrote:

To fix this situation up, you need to change the source IP address of the packets to fix them up. Unfortunately the only way I know of doing this is to use source-NAT on appropriate outgoing packets, [...]

You know what? Sometimes I'm too obscure for my own good. Since I tripped over this today, let me be explicit about what iptables rules I need to use, because this paragraph in my original entry led me to try to do this with SNAT alone, which doesn't work.

First, start with the basic dual identity routing policy based setup. Then I need:

iptables -t mangle -A OUTPUT -p tcp --dport 80 -j MARK --set-mark 2048
ip rule add fwmark 2048 priority 4999 table 11

iptables -t nat -A POSTROUTING -m mark --mark 2048 -j SNAT --to-source O

(Since this uses iptables marks to select what to act on, additional things to redirect can be set up with only an additional iptables rule to mark them. Also, this assumes that O is not the default route choice.)

We need all three pieces because SNAT alone won't (and can't) change the outgoing interface; the outgoing interface is set by the time the packet goes through the POSTROUTING chain (as the chain's name says), and SNAT can only be used there. Thus the first two lines will route the outgoing packets properly but with the wrong origin IP address, and then SNAT fixes the origin IP address up without altering the routing.

(If you just use SNAT, you get packets with the right origin address going out the wrong interface. If these packets still get to the destination, you might not notice this for a while.)