2007-10-30
The problem with big systems
In system design, we're periodically confronted with a choice between building our environment out of a few big systems or a swarm of small ones. For example, consider storage; if you want 48 Tb of storage, you could buy two Sun Thumpers or six or seven smaller 15-disk units. Big systems are often attractive; their very size gives them economies of scale (and often means they cost less overall), and it's easier to deal with only two boxes instead of six or seven.
(Here I am talking about big systems that are still cost competitive with a swarm of small systems, where you are not paying a markup for their sheer size.)
The problem with a big system, like a Sun Thumper, is that it is very expensive to expand. Not so much because the big system has a high markup for the pieces (although some certainly do), but because the units themselves are so big; if you buy your storage in units of 'one Thumper', adding disk space is relatively expensive since you have to buy another Thumper. Swarms of small systems are much cheaper to add to, because the individual units are cheaper.
How much you need this expansion depends, of course, and may tie into other system design issues. If you do not have to design for future expansion, you might as well just buy whatever is the cheapest way to get what you need right now.
Sidebar: on Thumpers
I'm aware that I'm exaggerating about Sun Thumpers. Part of the cleverness of ZFS is that it gives Thumpers a relatively cheap incremental expansion path; you can buy a mostly bare chassis, add disks a few at a time, and use easy raidz or raidz2 pool growth. If we didn't need at least three they'd be sort of tempting.