2012-11-03
Another go-around on the drawbacks and balances of automation
Today, Ben Cotton tweeted:
When I see a #sysadmin say "I don't use any configuration management", I mentally add "because I'm allergic to competence."
I have a complex reaction to things like this and after a small conversation I condensed some of my opinions to this:
@thatcks: My overall view on #sysadmin automation is that automation adds complexity as well as removes it. How each side balances out varies.
@thatcks: Before you automate, you manage your systems. After you automate you manage the automation (new work) and your systems (hopefully easier).
This deserves a bit more space than Twitter allows it.
All automation needs at least some management and adds some complexity. When you add automation, now you need to spend at least a little bit of time keeping the automation itself running and maintained; this was time that you did not need to spend before. This is ultimately because automation is software, not magic. But at the same time automation means that it takes less time and less work to maintain your systems (at least to the same level of quality), which is why people are driven to automation as their systems grow in size.
These two effects push against each other. How they balance out is not set in stone; it depends on the particular circumstances of your systems, on how much time managing the automation takes versus how much it reduces your system management workload. If you were previously logging in to a hundred servers to make changes on each, automation is going to reduce your workload a lot more than if you were doing this on ten servers. Or two. Similarly, automation that is a pain to keep going raises the gain threshold while easy automation lowers it. If your automation is so complex and fragile that you need a full-time person just to keep it going, you'd better be saving a lot of time with it (generally because you have lots and lots of systems). This degree of automation would be what we call 'overkill' for smaller systems, but when we say 'overkill' what we really mean is that it would take too much work to run for the system management gains it brings in practice.
The gains from improving system management generally scale based on how many systems you have, how often you change them, and how complex those changes are. Large, constantly changing environments are the best case; small static environments are the worst one. The cost of automation varies widely but generally not on any predictable basis, since a great deal depends on both software quality and the fit between the software and what you need to do. The small exception is that automation systems often have a target environment size that they're designed for and using them outside of a right-sized environment is extra-expensive.
(This goes both if you use a large-environment system in a small environment and if you try to make a small-environment system work in a large one.)
Sidebar: apples to apples comparisons with manual management
Some people will say that automation is automatically superior to managing your systems by hand because it insures that everything is repeatable and documented. This may be true in practice, but it is not required in theory; in theory you can have carefully and thoroughly documented (and tested) manual management processes. It just takes more time.
The flipside of this is that if you're going to talk about how little time managing systems by hand takes and thus how little automation would save you, you need to factor in this. It's only an apples to apples time comparison if you're spending that time to make your manual processes just as good as the automated ones.
(You can do this. We (mostly) do. But it undeniably takes extra time and work over plain ordinary manual system management, and it's easy to slip up.)