Combining dual identity routing and isolated interfaces revisited
Alas, I am now left with a mystery: according to the policy routing rules, it looks like a packet from IP1 to an address on that subnet should get routed via the gateway (and similarly for the other networks), [....]
You know what? I should have paid more attention to the mystery, because as it turns out such packets were getting routed via the gateway. I just didn't notice because I looked at the wrong thing when I wrote the original entry, and it worked most of the time; gateways are generally perfectly happy to accept packets for the local network and throw them back on the network.
(The one case where the gateway is not is when your gateway is also a firewall, and the firewall has filtering rules that wind up rejecting your forwarded packets. This is what happened to me yesterday, forcing me to look into the issue, although in retrospect some slightly odd things had been happening for a while. This goes to illustrate that you really should look into vaguely peculiar things, because they might be a sign of something important.)
To fix this problem we need to add an additional rule to each table from the original entry, more or less like so:
ip route add NET1 dev eth0 src IP1 table 20
(And similarly for the other two IP addresses.)
src IP1' bit is probably unnecessary (in fact, thinking about
it it almost has to be), but I wasn't sure when I set up my rules and
after this mistake I wasn't in any mood to take chances. So you get the
version of the rules that I am sure work, instead of a version that I
merely think should work.