A limitation in Linux's policy based routing

August 26, 2007

One of the more advanced things you'd like to do with Linux's policy based routing and a dual identity scenario is to be able to make more flexible decisions about what goes out what interface. Consider the case where you have two internet connections, one slow but reliable and the other one fast but currently flaky, and you have a different IP address on each. You would like to send not so important traffic (such as web browsing) over the fast but flaky connection while still having important traffic like your ssh sessions go over the slow but reliable one.

(Why yes, my DSL is being flaky at the moment.)

In theory the way to do this is simple: you use iptables to put a mark on whatever packets you want routed explicitly, and then you use ip rule to set up rules that route explicitly marked packets out the corresponding interface.

(Alternately you use marks to classify packets, so port 80 traffic would get the 'http' mark, and then set up routing rules to declare which way any particular class of packets was supposed to go.)

However, this doesn't work, or at least doesn't work the way you want. The problem is that by the time the packet passes through iptables to get marked, the kernel has already decided what source address it will have. If you put your mark-checking ip rules after your explicit source address based ones, they won't do anything; if you put them before, they will cause packets to go out the wrong interface for their source address.

To fix this situation up, you need to change the source IP address of the packets to fix them up. Unfortunately the only way I know of doing this is to use source-NAT on appropriate outgoing packets, which strikes me as inefficient and ugly for various reasons, and possibly sometimes dangerous.

(I can see why iptables behaves this way, since rules in the mangle OUTPUT chain may want to inspect the packet's source address. But it's still contrary to the documentation, at least in theory. It also implies that packets are probably going through the IP rules table twice, once before the mangle OUTPUT table, to assign the origin address and so on, and once afterwards.)

Comments on this page:

From at 2007-08-27 10:44:49:

Does that work well?

When I've had this exact problem [1], I've always just used "ip rule add from IPs" and had all the http traffic go through a proxy. The most annoying bit here is making sure all the http downloading software goes through the proxy, and it's also a good point for why bittorrent not having proxy support can be annoying :).

I'd be very suspicious of routing using port numbers, for instance I'm pretty sure that will screw up PMTU discovery. I've also seen more than a few sites that have links to http servers on special port numbers.

[1] Well in my case the DSL was "stable" but on a dynamic IP, while the modem had static IPs ... but basically the same problem.

By cks at 2007-08-27 23:05:38:

I've only tried doing this once, but it seemed to work acceptably for relatively oridinary use. I was willing to live with the 'port 80 is not all HTTP', since it gets most of the stuff I was planning to browse while my DSL was flaky.

I didn't see any PMTU issues, but then my PPP link has a lower MTU than my DSL and I was switching PPP to DSL, so at most I would have seen worse performance. I didn't look too hard at the speed at the time; my attitude was that it was still better than trying to fetch the typical large image-heavy web page over my modem.

The proxy approach is an interesting one that I'll have to remember, especially since I'm already using one to strip ads, cookies, and so on.

Written on 26 August 2007.
« Weekly spam summary on August 25th, 2007
On the naming of machines (part 2) »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun Aug 26 21:56:09 2007
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.