What we could use 10G Ethernet for in the near future

May 17, 2011

As I've written before, I don't think we're going to see any great use of 10G Ethernet in our immediate future; it's still too expensive to see it routinely start showing up on machines. But I've recently been thinking about what we could feasibly use 10G Ethernet for in the relatively near future, if (for example) we got money to buy a modest amount of 10G gear.

(I think that this is an interesting question in general. For the moderately near future, many places will be able to afford a few 10G connections but only a few will be able to afford a significant number of them. So the question is: what can you use a modest number of 10G connections for that makes sense?)

The nominally obvious place to deploy 10G Ethernet is in a fileserver infrastructure. Ignoring any lack of need for it, the problem is that when you have a SAN, increasing the speed of one network connection doesn't really help because you just wind up bandwidth constrained on other connections. We could give our fileservers 10G connections to the outside world but they'd only be able to use a fraction of it, since they only have two 1G connections into the SAN fabric. If we gave them 10G SAN links, the backends need fast connections or we max out at four 1G connections (at best). All of this adds up to a lot of 10G connections (and a lot of changes to our systems, but let's ignore that).

A lot of the other places that we could sprinkle 10G connections over have similar issues; isolated 10G connections between a few machines generally don't do us any good, because we don't have any super-hot, bandwidth constrained machines. This is in a sense the downside of building a balanced architecture; lifting it up requires improving a bunch of places at once.

However, there is one place that we could usefully deploy a small number of 10G ports. The core of our physical network is a set of switches chained together through pairs of 10G ports (each switch has two). This is an increasingly awkward architecture as time goes by (for reasons beyond the scope of this entry). We would love to move to a star model where there is a central 10G switch that all of the core switches uplink to; at this point even a four-port 10G switch would give us a useful simplification of the topology.

(Given an increasing use of 10G for switch to switch links, I suspect that this will be a major way that 10G works its way into most machine rooms. We can't be alone in having this topology problem.)


Comments on this page:

From 70.30.137.42 at 2011-05-17 18:54:03:

You state:

The core of our physical network is a set of switches chained together through pairs of 10G ports (each switch has two). This is an increasingly awkward architecture as time goes by (for reasons beyond the scope of this entry). We would love to move to a star model where there is a central 10G switch that all of the core switches uplink to; at this point even a four-port 10G switch would give us a useful simplification of the topology.

Having one switch in the core of your network is tempting fate to a certain extent as it is a single point of failure. :)

One product you may want to check out is the Juniper EX4500: it has forty 10 GigE SFP connections. While not cheap-cheap, it can get you started on a star topology. Of course ideally you don't want a single switch, so if you get a second (now or at a later date) you can chain them together in a "Virtual Chasis" system so that they appear as one switch to other devices:

http://www.juniper.net/us/en/products-services/switching/ex-series/ex4500/

You then have your other switches connect one uplink port to one of the (physical) EX4500s, and the other uplink port to the second physical EX4500: if one has hardware issues the other should still be fine (hopefully it's in the next rack or two over). You can then also use link aggregation (802.3ad) on the two uplinks since they're connecting to a single (virtual) switch to help increase bandwidth bandwidth. Similarly for any file servers that you may connect via 10 GigE: dual connections, one to each physical unit to help in redundancy and spread the load.

Of course if there's a software problem using the two virtually-link switches then it could still cause an issue, but one way around it could be simply powering down one of the physical chassises while the problem is sorted out.

I'm sure other manufacturers have simply virtual chassis technologies, but I'm not sure if they're available on units that are as inexpensive (relatively speaking) as the EX4500.

From 70.30.137.42 at 2011-05-17 19:00:37:

A similar thing exists on some of the products from Extreme Networks. They call it SummitStack:

http://www.extremenetworks.com/solutions/summit-stack.aspx
http://www.extremenetworks.com/products/summit-x650.aspx

Again, you can start with one unit and expand as you have budget to help reduce single points of failure if that's a priority.

Written on 17 May 2011.
« How our network is implemented (as of May 2011)
Why open source projects should use 'git rebase' or the equivalent »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Tue May 17 01:47:28 2011
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.