Wandering Thoughts: Recent Entries For 2007/03/28

A surprise to remember about starting modern machines

In the very old days you connected to Unix systems through serial terminals, which only had their getty processes started once init had finished processing /etc/rc.

In the old days you connected to Unix systems through rlogin(d), which was started through inetd, which was still started pretty much at the end of system startup.

These days you connect to Unix systems through sshd, which is often started relatively early in the system boot sequence. This means that you can easily wind up logging into a machine that hasn't finished booting, and conversely that just because you can ssh into a machine doesn't mean that it's finished booting.

This mistake was at the root of my debugging adventure today. We're switching to a new system of managing NFS mounts on our Ubuntu machines, and I was seeing a mysterious problem where the test machine would boot up with its NFS mounts partially or almost completely missing. Due to local needs we start sshd before doing our NFS mounts, which we have a lot of, so what was really going on was that I was logging in to the machine while it was grinding through the NFS mounts. Once I realized what was actually going on it was a definite forehead-slapping moment (although a reassuring one, apart from the wasted time, since nothing was actually wrong).

You can get into really weird states because of this. In the past I've managed to have init.d scripts hang trying to start something; if they run after sshd starts you could still log in to the system, poke around, and have everything look pretty normal (depending on what was left in the boot sequence). Except that things like reboot wouldn't do anything, because as far as init is concerned it was only part way through transitioning into a runlevel and it wasn't about to let you change to another one just yet. The whole experience can make you think that the machine is badly broken, because reboot doesn't complain and a machine that doesn't reboot on command is usually in serious trouble (often with things like kernel panics, unkillable stuck processes, and so on).

(I think what tipped me off back then was the same thing as this time around; I got a process tree dump and saw the startup script still running.)

Dual identity routing with Linux's policy based routing

In the last entry I talked about the idea that an end node machine in a VPN actually has two distinct identities that need to be routed separately: the inside identity that should be routed over the VPN, and the outside identity that should be routed over the Internet. In this case each identity has an IP address associated with it; call the inside identity's IP address I and the outside identity's IP address O.

We can do this sort of routing with Linux's policy based routing, which fortunately is not as scary as it looks. To simplify somewhat, Linux does policy based routing by having multiple routing tables, and then providing rules to pick between them. We need two new routing tables, one for each identity, but fortunately we don't need them to have many routes; all they actually need is a default route.

So we set up the tables like this:

ip route add default dev VPN table 10
ip route add default dev REAL table 11

(Here VPN and REAL are your VPN and REAL network interfaces; you may need to add 'via GATEWAY' to the one for your real connection. My setup uses PPPoE DSL, so I can just throw my packets in the direction of ppp0 and have everything work right.)

Now we need two policy rules so that traffic explicitly from one or another of our identities is sent out the appropriate interface:

ip rule add from I iif lo priority 5000 table 10
ip rule add from O iif lo priority 5001 table 11

(The table numbers and the priorities are mostly arbitrary.)

This says that traffic explicitly from I should consult table 10 before theoretically falling through to the default main routing table. Similarly for traffic explicitly from O, which gets sent to table 11. Since tables 10 and 11 have a default route, we never actually fall through.

The final thing to do is to set up which networks default to using the inside or the outside identity when we connect to them. We do this in the regular routing table by choosing which interface we send a route to, for example to route 128.100.5.0/24 over my VPN connection I do:

ip route add 128.100.5.0/24 dev VPN

(Unless you change your default route, the default identity is your outside identity and thus you just need to add route entries for stuff you want to go over the VPN. Because the most specific route matches, you can make some large network go through the VPN, and then exempt specific subnets if you need to.)

This works because both the VPN and the REAL interfaces have local IP addresses associated with them, which determines the local IP address for outgoing connections that haven't explicitly specified it. (Connections that have specified the local IP address will get stolen by our explicit rules before we reach the main routing table.)

As a bonus you can make outgoing connections use a specific identity by specifying the origin address; for example, I could still connect to a machine in 128.100.5/24 as my outside identity by using 'ssh -b O <whatever>'. (This might be necessary if the VPN connection dies for some reason.)

(I owe a real debt to the documentation for the ip command, which turned out to be quite understandable once I actually sat down and read it carefully. There's a lot of useful and interesting stuff there.)