Getting your networks to your racks

October 12, 2007

I'm in the process of getting a new test server installed in a rack in our machine room. This means we needed to set up some network connections for it, which involved someone dragging yet more cable around our machine room and finding switch ports to plug into to get the necessary networks (which turned out not to be an entirely trivial thing).

As a result, the whole thing got me thinking about the issue of the best way to get all of your networks to where they need to be in your racks. The good approaches that I can think of right now are:

  • run each machine's network cables out of the rack to central switching points. But if you do this, you can have more and more cables snaking under the floor, and part of having neat racks is running as few cables as possible out of each rack.

  • one ordinary gigabit switch per rack, with all the servers connected to it and as many networks as you need brought into the switch as VLANs. But then the entire rack is sharing a 1 gigabit uplink into your core fabric, which may not be enough.

    (I don't think that trunking is a good workaround, because it will consume switch ports in your core fabric at a ferocious rate.)

  • one switch with a 10 Gb uplink per rack. This is costly, especially at your central switching points (switches with more than two 10 Gb connections are apparently really expensive).

The first approach can deliver the most efficient network bandwidth, because you can clump machines that all need to talk to each other together on one switch, regardless of where they are physically located. You can't count on clumping such machines together in a single rack for at least two reasons; first, you may have more than a rack's worth of such machines, and second, machines may need bandwidth to more than one such clump.

Our current approach in practice is mostly the first option, partly because almost all of our new racks are only one row over from our network core. We have one new rack that is all the way on the other side of the room; fortunately it doesn't currently need more than a gigabit in total and its machines are all on one network. (And the rack is nearly full of modern machines, so the latter is unlikely to change for several years.)

Written on 12 October 2007.
« A gotcha with command order in pipes
Weekly spam summary on October 13th, 2007 »

Page tools: View Source, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri Oct 12 23:17:46 2007
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.