In an earlier entry, I mentioned that firewalls are a classical case of difficult testing where differences between your test and your production environments can be vitally important. Let's elaborate on that.

Suppose that you have some firewall rules changes that you want to make. As a good developer-style sysadmin, you are not going to just dump them on your production firewall; instead you have a test firewall that you push rules to first for testing. But here's the question: how is your test firewall's networking configured, specifically, do you give it test IPs and networks, or do you configure it exactly identically to the production firewall, using the production firewall's IPs and networks?

If you give it production IPs and networks, it obviously has to be completely isolated from your production environment. In turn this means that it needs to have its own supporting (and testing) network infrastructure (with multiple machines, network connections, etc), and you have to somehow push configuration updates into that test network infrastructure.

(I'm going to assume that our only concern is testing firewall rules changes; we're going to assume that things like firewall monitoring systems continue to work fine, so we don't have to build something to test them inside this isolated environment.)

If your test firewall uses test IPs and networks, it doesn't have to be completely isolated from your production environment and can reuse a bunch of your existing update and management infrastructure. This sounds good, but there's a problem: errors in IP addresses and network blocks are exactly one of the problems with firewall changes, yet you can't test for these errors if your test firewall uses test IPs and network blocks. Your test version of the change, using test IPs, can be done right, yet you've made a mistake when writing out the production IPs; you'll only find out when you push the update to the production firewall and things start breaking.

(So what differences between your test and production environments are acceptable to have? My only thought right now is that differences in things that you don't change seem safe, because then you can verify all of those differences once and know that things are good from then onwards.)