Chris's Wiki :: blog/sysadmin/AutomationBadFitHere

What's in the way of us using automated configuration management

June 18, 2013

Every so often I poke at Puppet or Chef or one of the other automation systems and consider if we could really use it. And every time I do I find it a hard sell, even to myself (much less hypothetically to my co-workers). Today I've decided to try to write down my collection of technical reasons (to go with other reasons):

We already have a scripted install process and build instructions for all of our machines, and also more or less scripted package updates.
We install almost nothing on our machines other than vendor packages or things scripted as part of the install process; we don't have applications that must be deployed to random machines.
It's quite uncommon for us to add even vendor packages to our machines after their initial deployment. It generally only happens when users want some additional Ubuntu package on the login or compute servers, which is pretty rare.
Many of our machines are singletons, where we only have one of the particular sort of machine; one external MX gateway, one central mail server, one print server, and so on. The few machines that are significantly duplicated naturally tend to have the most automated install process.
(Installation of login or compute servers is basically completely automated. Given racked hardware we can have a new one up and running about as fast as it can unpack many, many Ubuntu packages on to its disks.)
We (re)build machines only very occasionally; deploying a new machine is a rare occurrence. We don't get new hardware in very often, we have few enough machines that they almost never break, and we only 'upgrade' OS versions at most every few years as new Ubuntu LTS releases come out.
We consider it a feature that it takes manual steps to deploy a change to a singleton machine. To put it one way it acts to insure that you're paying attention to how your change actually works.
We have our own automated distribution mechanisms for things like the passwd file; these would have to be coordinated with or hooked into any general automation system.

The short version is that we've already automated most everything we commonly do to more than one machine. I don't seem much room for an automated configuration management system to come in and do new things for us; this means that it would have to replace existing, already developed and working automation. There's some benefit to using standard tools for automation but I'm not convinced that there's a lot.

It's possible that I'm missing things that Puppet, Chef, et al could do for us because the usual examples I read are bright cheerful 'let's deploy a canned web server configuration on to a random machine' ones and we don't have that problem. In a chicken and egg problem, I can't find the energy to read the documentation for Puppet because I suspect that I won't find anything we can use.

Comments on this page:

From 91.198.246.131 at 2013-06-18 04:24:31:

Actually, there's also a templating system. It allows you to provide few templates of configs and then deploy them filled accordingly to what on a specific machine is needed.

Apart from that, I don't think there is a lot of selling points for your environment. CFEngine or Puppet are mainly useful when there are many changes to the infrastructure: new servers, new services, reconfiguration, services and functions moved around.

In smaller scale CFEngine or Puppet do pay for themselves, but the gain is not big enough to justify changing from already-deployed automation (on the other hand, is certainly big enough to deploy them if no automation was used previously).

You might want to look at CFEngine 3.x (if you haven't done so yet). It's a declarative programming language. IMHO it's more obviously a programming language than Puppet (Puppet looks to me like a thing that only mimics one), and it's a declarative one, unlike Chef (which is basically a Ruby framework, not a language on its own).

BTW, you could use CFEngine or Puppet not as a standalone configuration management tool, but as a part of your scripting. In the long run, it could smoothly and gradually replace other tools, as it proves to be useful and as you learn it. Or it could be easily withdrawn, if you find it not that useful at all. But, of course, this is not a selling point for the tools.

-- dozzie

From 128.210.189.101 at 2013-06-18 09:18:40:

Wrong. You're already using automated configuration management, it's just a home-grown system that doesn't provide ongoing support.

If it meets your needs and the costs of switching to something new are greater than your current costs plus technical debt, then keep on truckin'.

From 128.101.135.18 at 2013-06-18 11:38:46:

I am working on my second passage through the "home grown" to "external/open source" configuration management system (different companies). Getting the energy to make the transition is hard. It also requires a large time and effort investment.

But it was freeing in ways I did not anticipate.

After the first transition, I was more free to experiment with new software and architectures. Less time was spent understanding the automation (written by a mad genius). Less time updating our automation scripts. More time solving problems once (and only once). I started to work through my system logs and solve the little unimportant issues, which solved more annoyances for my users than I was previously aware of. Backups transitioned to capturing only user data and puppet scripts. No longer did I need to capture the essence of any one system. Coordinating with other Admins also became much easier.

From 24.165.55.105 at 2013-06-20 21:02:56:

I was slowly migrating from nothing to Chef at previous job. Any new server built from ground up was done using Chef, including our one-shot servers (most of them are one shot boxes)

There are a few advantages from it:

1) Infrastructure as Code: It's a cliche phrase but I can easily see what each server is doing by what recipes are applied to it, and any incoming sysadmins can quickly see the key configuration details without having to play guessing games "Is this relevant? Is this?"

2) Enforced Consistency and Change Management: With every box picking stuff up from chef on a scheduled basis, changes to important functions are automatically set back to what they should be, rather than someones fiddle or tweak. All the chef cookbooks are plain files, so they're stored in git. Every change has an associated git commit, so I can easily see why something was changed on a box. That means co-workers and I are less likely to be tripping over each other, plus is really useful to show when being PCI-DSS audited.

3) Easy replacement: If a box dies, it's quickly replaced, functions easily moved around infrastructure without hassles.

Written on 18 June 2013.

«	My job versus my career: some thoughts

Our approach to configuration management