Wandering Thoughts archives

The limits of isolated interfaces

So you've carefully isolated your management interfaces, making sure that packets can't leak from one side to another because they literally can't see the other side. You're safe, right?

Well, not quite. While an attacker can't get their own packets across to your management interface, they may still be able to trick things on your machine into generating outgoing packets for them. There's a number of cases:

• packets with a management origin to your management IP: this should be handled by interface isolation, which makes the outside interfaces pretend that the management IP isn't a reachable IP.

• packets with a management origin to your outside IP: if your programs respond with the origin IP that they were contacted on, everything is fine; the outside IP forces the packet to use the outside routing table, which does not include your management network, which means that they fail to route.

  (TCP connections automatically behave this way, and my impression is that many but not all UDP-based programs behave this way because it makes life happier on multi-homed machines.)

• your programs are willing to send packets or initiate connections to an IP address that it gets from the payload of a message (for example, XDM works this way). This is trouble, because not even a firewall can stop these and programs on your system have access to all of the interfaces, including the management one.

(If you have things so isolated that only some programs can talk to the management interface, you've gone well beyond mere interface isolation.)

Just to make life more interesting, this exposure exists whether or not the machine actually routes packets; it is enough for it to be multi-homed on your management network and for an attacker to know it.

This doesn't let the attacker get arbitrary packets to your management network; depending on what they're tricking, the packet contents are likely to be highly restricted. Still, it's useful to remember that interface isolation is not a cure-all.

The appeal of GNU tools

I admit that every so often the freakish superintelligence of the GNU versions of programs has its appeal. Consider, say, units:

You have: 7 feet + 2.5 inch
You want: mm
        * 2197.1
        / 0.00045514542

GNU date is another good one:

; date -d 'last sunday'
Sun Apr 13 00:00:00 EDT 2008

Or even better:

; date -d 'last saturday -1 week'
Sat Apr  5 00:00:00 EDT 2008

(Judging from the manpage, BSD date may also have some freaky superintelligent date math stuff, and judging from some experimentation, GNU date is not all that superintelligent, or at least not that well documented, since I couldn't see an obvious way to do something like 'the third Saturday in January 2008'. As an aside, the GNU date documentation perfectly illustrates the downfall of the GNU's (text)info documentation format.)

Also, I will admit that sometimes GNU is plain right about things. For instance, I'm pretty convinced that the -h option to df, du, and ls just is easier for people to read; I find myself usually using it unless I am going to feed the output to something else.

Management interfaces as isolated interfaces

Setting up isolated interfaces can seem like a lot of work that has relatively little point, but there is at least one important case for them: management interfaces on machines that route traffic.

Consider your typical garden variety routing firewall. Let us assume that you do not want to make it directly accessible on its outside router IP, because then the outside world could attack it, or any of its normal inside IPs, because then compromised machines inside your network could attack it. So you give it a connection to your management network.

You don't want people who route through the firewall to be able to reach your management network; even if nothing on the management network could route packets back, simply flinging packets at machines can do a certain amount of damage. (Especially when any number of things on your management network may have less than completely robust IP stacks and software.)

You could protect the management network with firewall rules alone, but this is at least somewhat brittle and there's a bunch of cases to think of, where one slip with one network could leave holes. Setting up the management interface as an isolated interface gives you another solid layer in the way of things going wrong. Flub the firewall rules one day? You're still not leaking packets.

(It is a pity that it is so hard to set up isolated interfaces, or equivalently to divide interfaces into interface groups. But then I suppose it's not something that most systems do often.)

Get statistics

Here is one of the best things you can do to improve your ability to find and fix problems:

Get statistics.

Over and over I have seen statistics be a big part of solving problems. Not infrequently they are a big part in identifying the problems; sometimes having a reliable way of telling whether the problem is happening or not is half the battle.

(There is a world of difference between 'sometimes the NFS fileservers are slow' and 'sometimes the new RAID controller has average IO service times of over a second'.)

You shouldn't just get statistics when you're having problems; you should get them all the time and then keep them as long as possible (ideally forever). Having long term statistics is an even more powerful weapon, both because it lets you answer more questions and because it lets you compare what is different between the past (when you didn't have the problem) and today (when you do).

Unfortunately all of this is easier said that done, as very few systems come with good setups for gathering and keeping this stuff. In many places, building and maintaining the necessary tools just can't compete with other, more immediately urgent priorities.

(I confess that ours is one of those places.)

Sidebar: don't archive summaries

It is tempting to periodically summarize your stats and then archive only the summaries. While this is better than nothing, you really want to archive the full information. If you only keep summaries, you are counting on only needing the information in the summaries; in other words, you are assuming that you already know what information you will need to know to fix future problems.

This is not an assumption that I would make, to put it one way.