A limitation in Linux's policy based routing

One of the more advanced things you'd like to do with Linux's policy based routing and a dual identity scenario is to be able to make more flexible decisions about what goes out what interface. Consider the case where you have two internet connections, one slow but reliable and the other one fast but currently flaky, and you have a different IP address on each. You would like to send not so important traffic (such as web browsing) over the fast but flaky connection while still having important traffic like your ssh sessions go over the slow but reliable one.

(Why yes, my DSL is being flaky at the moment.)

In theory the way to do this is simple: you use iptables to put a mark on whatever packets you want routed explicitly, and then you use ip rule to set up rules that route explicitly marked packets out the corresponding interface.

(Alternately you use marks to classify packets, so port 80 traffic would get the 'http' mark, and then set up routing rules to declare which way any particular class of packets was supposed to go.)

However, this doesn't work, or at least doesn't work the way you want. The problem is that by the time the packet passes through iptables to get marked, the kernel has already decided what source address it will have. If you put your mark-checking ip rules after your explicit source address based ones, they won't do anything; if you put them before, they will cause packets to go out the wrong interface for their source address.

To fix this situation up, you need to change the source IP address of the packets to fix them up. Unfortunately the only way I know of doing this is to use source-NAT on appropriate outgoing packets, which strikes me as inefficient and ugly for various reasons, and possibly sometimes dangerous.

(I can see why iptables behaves this way, since rules in the mangle OUTPUT chain may want to inspect the packet's source address. But it's still contrary to the documentation, at least in theory. It also implies that packets are probably going through the IP rules table twice, once before the mangle OUTPUT table, to assign the origin address and so on, and once afterwards.)