Wandering Thoughts archives

The advantages of separate machines for separate things

Sometimes it seems that system administration goes in cycles. Right now the cycle is moving back towards consolidation of services on fewer machines, so I want to talk about the advantages of using separate machines (whether virtual or physical) for different services, instead of putting them all on the same machine with various degrees of clever tricks.

(The genesis of this entry was a comment on this entry, talking about how one could use one machine instead of two to do the job I was tackling.)

First off, it is usually simpler to configure the machines. This especially so if you need two instances of the same system, such as a mailer or a web server, as many system setups are simply not designed with this in mind and require a bunch of changes to work right (and missing a change can be problematic). Running only one system instance of a service is the common case that everything is designed for, so you might as well go with the flow; it's easier.

Second, it gives you isolation and independence; when you do things to the underlying system environment, it affects as few services as possible. The obvious case is taking a machine down or rebooting it, where if you have a bunch of services on the machine you need a time that is acceptable to have all of them down (at once). Similarly, if you're planning an OS upgrade or change you need to have all of the services ready to go on the new OS instead of just one of them (and you can't do a split upgrade, keeping some services on the old OS version and moving others to the new one).

This also implies that your one machine needs to be configured for the union of what all of its services require. This sounds abstract, so I'll give a concrete example: do you need to mount all of your NFS filesystems? Our main mail machine has to have user home directories mounted from our fileservers and use our full /etc/passwd file, but the spam forwarding machine does not. As a result, the spam forwarding machine is almost entirely decoupled from our fileserver infrastructure.

(Our fileserver infrastructure does much more than just plain NFS service, but that's another entry.)

And of course you get fault isolation; if something goes wrong on one machine, it only takes down one service instead of a whole bunch of them. Here, things going wrong can be anything from system crashes to CPU fan failures to someone accidentally nudging the network cable out when they were in the machine room doing something else.

Sometimes the services really are so tightly coupled that you would never use the freedom that one service per machine isolation gives you. But in my experience this is the rare case; far more common is situations where some services are easier to interrupt than others.

Virtualization is not a cure-all for these issues; if anything it can make some of them worse, because it can concentrate a lot of machines and services onto one physical piece of hardware and one host OS. You can do better, but it gets expensive.

(From personal experience, doing anything to the host machine for a bunch of virtual machines is a pain in the rear if you don't have easy failover.)