Why office switches plus VLANs aren't the answer for sysadmins
In more or less reply to my last entry on wiring offices for sysadmins, I got a tweet:
@thatcks 2 drops is enough. Desk switch per sysadmin + vlans and bob's your uncle
(I can't find the tweet in the person's public stream right now so I'm not going to attribute it. Maybe I should paraphrase it instead of quoting directly, but any paraphrase would probably be longer.)
I thought of this, but there are three reasons why this doesn't work: uplink bandwidth, maintenance overhead, and a network design that doesn't have everything as part of the same VLAN fabric.
The maintenance overhead is pretty straightforward. Any time you want to set up a new network or tear it down, you need to modify a bunch of switches to add or delete VLANs; each sysadmin's switch and then your master uplink switch for all of the sysadmin offices. Even if sysadmin office uplink is the only thing this switch does, it is a 'production' switch in that other sysadmins are not going to be happy if something goes wrong on it and things suddenly drop off the network. This is a pain at the best of times and can rapidly reach the point where running cables around the office is easier.
The uplink bandwidth issue is twofold. First, your total bandwidth across all VLANs is limited by the switch uplink bandwidth. In all realistic configurations this means you have a 1 GB total limit, and yes this can easy get in the way of certain sorts of testing. Second, the more VLANs you push over a single uplink, the more bandwidth you lose to background chatter on those VLANs and any ordinary traffic your regular machines may be doing (on regular production VLANs). Among other things, this complicates efforts to measure the true network performance of machines; are you getting less than a gigabit because they just can't deal with a gigabit, or is it because of other traffic on your office switch?
You can deal with some of the uplink issue by not propagating currently unnecessary VLANs to your office switch, but then you wind up needing to reconfigure at least your master switch every time your VLAN needs change. We are currently in this situation and take it from me, it is a pain in the rear that discourages testing.
The network design issue is that some of your networks may not be designed to run over your normal VLAN fabric and switches. There are three examples of this here (for background, details of our network setup):
- we have several port isolated internal networks that are kept out of our regular VLAN fabric except at touchdown points for internal firewalls.
- we have a completely isolated management network that runs over its own dedicated switch fabric. Bringing it into contact with our regular VLAN fabric in order to get it to our office is at least a violation of its design and has potentially bad effects if, for example, some of the switches involved also have management ports that are directly connected to the management network.
- we have deliberately isolated, non-VLAN'd, non-connected iSCSI networks. We may at some point want a port on an iSCSI network in our office, but we definitely do not want to pull the iSCSI networks into our VLAN fabric; we want a direct port to the iSCSI switch.
Trying to merge together otherwise isolated networks and VLANs on a subset of switches makes me nervous. There is a real possibility for accidental leaks and contamination (as well as weird side effects), and it's especially acute when you're reconfiguring the master office switch often. Of course this also holds for putting new VLANs for test networks on the master office switch, since these new VLANs are not part of your regular VLAN fabric and should never be propagated to it.
Sidebar: what I want in an office setup
The short form version is what I want is one port to carry the primary networks for my main machine, one port for a switch with all of our regular VLANs on it so I can connect to them for testing, one port with our port isolated network for user machines, one port for our isolated management network, and several other ports that I can connect to anything as the need arises. This is at least five ports.
(Thus I think the basic need is two ports for static stuff (one for your primary machine, one for the 'has everything' VLAN switch), plus some number of ports for floating things.)
Right now I have, effectively, three ports: one port with a selection of our regular VLANs that feeds through to my main machine and one port with an Ethernet splitter that gives me our port isolated network for internal machines plus our management network. The latter two networks only run at 100 Mbits each but that is not currently a problem. This is not enough ports, and I don't do as much networking work as my co-workers.