A stupid switch configuration trick
Today we discovered that we'd accidentally done a stupid switch trick to cheat ourselves out of some link bandwidth: we'd accidentally configured some untagged ports on one of our top-level switches to also carry some tagged VLANs.
(Normally you configure switch ports in one of two ways: either they carry one untagged VLAN and nothing else, or they carry some tagged VLANs and no untagged traffic.)
This is effectively invisible, because the regular traffic is still flowing, and nothing complains about the extra stuff. These days switches generally drop tagged VLANs that they're not expecting, and hosts just ignore the strange traffic just like they ignore all the other weirdness that is probably floating around your network.
(Every so often I run
tcpdump on my networks, just to see what sort of
broadcast traffic and misdirected stuff turns up. It's usually good for
some nervous amusement.)
This can be an annoyingly easy mistake to make, because many switches don't have a per-port summary of VLAN membership that lets you see at a glance all the VLANs that a port is part of; you have to infer a port's VLAN membership list by going through each VLAN and looking to see what ports are members. This is especially irritating when you have a lot more VLANs than ports, as we generally do.
(Most of our VLAN-aware switches have been configured with all of the VLANs we have, even if they don't currently carry traffic for that VLAN.)
We think that the mistake crept in when we converted the ports from carrying various bundles of tagged VLANs (and no untagged traffic) to being untagged touchdown points for particular VLANs. Since you can't set or clear a port's entire VLAN configuration in one go, you have to go through each VLAN and remove its tagged membership; if you miss some, you get what we wound up with.
To be honest, I suspect that we weren't losing too much link bandwidth to this, although it depends on just how noisy some of our networks are. And we'd only have lost outbound bandwidth, since the ports are only being sent extra traffic, they're never sending any back.
(This was not a security exposure because we control everything plugged into the switch and switch ports in question.)