Wandering Thoughts archives

An example of the evolution of a real network

Our network as it is today has a little oddity in how it is implemented. There is a set of separate switches for one particular private sandbox network, which is odd enough considering our normal network implementation transports all sandboxes over a set of backbone switches. Even more oddly, these switches are not generic dumb switches, because they transport the sandbox between themselves as a tagged VLAN (and detag it to edge ports). All of our printers are connected to this sandbox and to this single-subnet switch network.

To understand this oddity, we need to walk back in history.

The switches are all VLAN-aware and the inter-switch links are tagged because this set of switches used to transport several different networks, not just one. In fact, it used to transport several of our public networks, which, yes, our printers were connected to.

Our printers used to be connected to our public networks because back in the old days, we did not have and use private subnets; all of our IP address space was public and everything went on it, printers included. The printers stayed on our public networks when we migrated other things off because renumbering printers is a huge pain in the rear (and we've had mysterious problems with it in the past).

Well, fine. But why did the printers have a completely separate network of switches, instead of just having their traffic transported over our core backbone switches?

The answer is: because those public subnets weren't (and aren't) carried on our core backbone switches. And in turn, this is because how they were configured on our previous generation of switches. On our previous generation, our big three public subnets were bundled together into what the old switches called a 'protocol based VLAN'; the switches could peer into the packets to pull out the subnet address and treat it as basically a sub-VLAN. This let us configure most ports as 'VLAN BLUE, but only subnet 128.100.X' and some ports as 'VLAN BLUE, all subnets'.

Our current-generation switches do not support this sort of VLAN. For various reasons we decided that they should carry only one of the three subnets in the old BLUE VLAN, the one we were moving all of our servers to (and that did not have any printers, at least once we did a bit of work). This meant that the printers needed their own network infrastructure to get the other old public subnets to them.

So there you have it. An odd and otherwise inexplicable bit of our network infrastructure is the way it is because of decisions that were probably made ten years ago and a completely logical series of decisions made since then.

Sidebar: one reason to carry only one subnet in the new world

My memory of this has been blurred by the passage of time (and by being only peripherally involved in this set of decisions), but as I remember it one reason that we did not want to carry all three subnets on our new backbone switches is that it would have required inventing new VLAN numbers for at least two of the three subnets. This would have led to a confusing situation where subnets were in different VLANs (with different VLAN numbers) depending on which bit of network gear you were looking at.

OpenBSD pf rule numbers and tcpdump filtering

OpenBSD's pf packet filter can log packets to a pflog virtual network device if you ask it to; in normal configurations such logged packets appear on pflog0 and then wind up saved in /var/log/pflog. Around here we log most rejections and some stuff that we accept, partly because this makes tracking down firewall issues much easier.

If you use 'tcpdump -e' on a pflog device or log file, it will print the rule number of the rule that caused the packet to be logged (or possibly the last rule number, I'm not sure) as part of the output. It will also try to print some information about the rule itself, but I suspect that this can become inaccurate for saved log files if you add, delete, or reorder rules in your pf.conf.

(Well, if you use OpenBSD's tcpdump. I doubt anyone else's version understands this stuff.)

In theory you can use tcpdump to filter by rule number, with the 'rnr' or 'rulenum' filtering condition. In practice this is tricky to use because of two issues. First, you need to determine the number of the pf rule that you want to match against, which is not as simple as counting rule lines in your pf.conf. The authoritative source of rule numbers is to use 'pfctl -vv' when dumping either the filter rules or the NAT rules (and who knows, maybe queue rules too). You want the lines that start with '@'; the number after the '@' is the rule number. Thus:

pfctl -vvsn | fgrep @
pfctl -vvsr | fgrep @

If you do this you will immediately discover the second issue: separate sorts of rules number their rules independently. I believe that block and pass rules share one sequence, and then nat, rdr, and binat rules are each numbered separately; I don't know about scrub rules. (This is what we see on our OpenBSD firewalls, at least; on some of them, we have four different rules that are all 'rule number 0'.)

You can restrict which sort of rule you are matching with the filter 'action <TYPE>', eg 'action rdr and (rnr ...)'. Helpfully (or not), you can match blocks and passes separately.

On the whole I think that it's easier to match with IP addresses and the like, and I expect this to be more robust for looking through old logged data. If you have to keep extra logging that is making this verbose, action filter rules may help.

(One of the things that may make this necessary is that, to the best of my understanding, rules that rewrite packet addresses log the post-rewrite address. Thus, using 'action rdr and ...' can help you see only the packets that got rewritten to an active host's address, instead of all traffic involving that host's regular address that you're logging.)

Sometimes having a system programmer around is the right answer

In SystemProgrammerDanger, I wrote about the danger of having a system programmer around (to summarize, we automatically reach for things like the source code). But there's a flipside of that, handily illustrated by my recent entry on getting stale file errors for local files; sometimes the system programmer approach is the right one.

When my colleague brought up this odd issue he was having with local files giving 'stale filehandle' errors, my first reaction was to grep through the kernel source for places that returned ESTALE errors; I figured that there couldn't be too many of them, since ESTALE is a very specific error (I was mostly correct about this). Reading through the code the grep found soon pointed me to the likely issue, especially once my colleague also reported that the system had disk errors and he'd been seeing odd stat() results for other files. All of this took me only a few minutes (partly because I already had kernel source available).

I'm pretty sure that this was the fastest way I could have found the answer. And I found it by taking a system programmer's path.

(Okay, web searches do suggest that other people have run into this before and have identified it as being caused by disk corruption and sometimes fixed by fsck. This is decent operational advice but doesn't tell you what's really going on. My personal view is that knowing what's really going on is important because it gives you confidence that you've dealt with the real problem instead of just papered over a symptom.)