== A limitation in Linux's policy based routing

One of the more advanced things you'd like to do with Linux's [[policy
based routing DualIdentityRouting]] and a dual identity scenario is
to be able to make more flexible decisions about what goes out what
interface. Consider the case where you have two internet connections,
one slow but reliable and the other one fast but currently flaky, and
you have a different IP address on each. You would like to send not
so important traffic (such as web browsing) over the fast but flaky
connection while still having important traffic like your ssh sessions
go over the slow but reliable one.

(Why yes, my DSL is being flaky at the moment.)

In theory the way to do this is simple: you use _iptables_ to put a
mark on whatever packets you want routed explicitly, and then you use
_ip rule_ to set up rules that route explicitly marked packets out the
corresponding interface.

(Alternately you use marks to classify packets, so port 80 traffic
would get the 'http' mark, and then set up routing rules to declare
which way any particular class of packets was supposed to go.)

However, this doesn't work, or at least doesn't work the way you want.
The problem is that ~~by the time the packet passes through _iptables_
to get marked, the kernel has already decided what source address
it will have~~. If you put your mark-checking _ip_ rules after your
explicit source address based ones, they won't do anything; if you put
them before, they will cause packets to go out the wrong interface for
their source address.

To fix this situation up, you need to change the source IP address
of the packets to fix them up. Unfortunately the only way I know of
doing this is to use source-NAT on appropriate outgoing packets,
which strikes me as inefficient and ugly for various reasons, and
possibly sometimes dangerous.

(I can see why _iptables_ behaves this way, since rules in the mangle
OUTPUT chain may want to inspect the packet's source address. But it's
still contrary to the documentation, at least in theory. It also implies
that packets are probably going through the IP rules table *twice*, once
before the mangle OUTPUT table, to assign the origin address and so on,
and once afterwards.)