The most interesting reason for an unrouted sandbox network

May 15, 2011

I recently wrote up our network layout and mentioned that while most of our private 'sandbox' networks are routed internally, we had a few that are unrouted for various reasons. This is the story of the most interesting unrouted one.

We have a number of firewalls. Because we are cautious people and these are very important systems, we have hot spares. Now, there are at least two ways to handle hot spare firewalls; you can have them physically plugged into the production networks but with their network interfaces not configured with the live firewall IP addresses, or you can have their interfaces fully configured but not have cables plugged in. We've decided that we prefer the second approach for various reasons, including that it avoids certain sorts of bad accidents.

(This does mean that we have to go to the machine room to swap in a hot spare, but in practice we consider this not a serious limitation. Among other things, we almost never have to do that.)

Since we're sane sysadmins, we don't edit firewall rules on the actual machines; instead, we edit them on a central master machine and then push the update to the firewall (with an automatic reversion if there are obvious problems). Of course, to keep the hot spares ready to go at a moment's notice we need to push the configuration updates to them too.

So: what management IP address does a hot spare firewall get, and on what network? It can't be an address on our public networks, because hot spares are fully configured with the gateway public IPs and so their network cable can't be plugged in lest they and the live firewall have a big fight over who owns those IP addresses. It can't be an address on any of our routed sandboxes, because then the hot spare for the firewall that provides routing for that sandbox can't be plugged in for the same reason.

Our conclusion was that they had to be on a new unrouted sandbox. This sandbox has their management interfaces and the central master machine that updates are pushed from, and if we need to log in to the firewalls (which we do every so often) we do it by indirecting through the central machine.

(OpenSSH has some stuff that make this indirection painless and effectively invisible with the right setup.)

Written on 15 May 2011.
« Our environment illustrated: what network cable colours mean what
How our network is implemented (as of May 2011) »

Page tools: View Source, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun May 15 01:14:39 2011
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.