The long term future of any particular configuration management system

September 27, 2013

From my perspective, one of the problems with the current profusion of configuration management systems is exactly that there is a profusion of them. I don't have a problem with choice per se (or having to sort through them all); for me the issue is the uncertainty that this creates for the long term future of any particular system.

Right now we have a lot of CM systems; off the top of my head I can think of CFEngine, Puppet, Chef, Ansible, and SaltStack, and there are probably others. It seems unlikely that they will all survive as actively developed and used systems over the next five to ten years. Some of them certainly will but probably not all (and I kind of hope that not all of them survive, because if they all do that's a lot of duplicate or near-duplicate effort).

Unless you're confident in your ability to pick winners and losers here (which I'm certainly not), the net effect of this is to add more uncertainty about the long term costs and payoffs of using any particular current CM system. If today you pick what turns out to be a loser in five years, well, that's all the payoff time you're getting (and now you'll have the costs of switching what's likely to be a well-developed environment to another CM system). At least for me this drives up the short term payoff I want to see from a CM system before it seems worthwhile to switch to it and I keep having problems seeing that.

(I don't have a good answer to this issue. I'm not sure there is one. And certainly my deep-seated skepticism that a CM system wouldn't be a big win here is part of the general issue; if I thought a CM system would be a significant improvement, well, I'd have that short term payoff. Instead I'm left feeling that I'd get at best a small incremental improvement over the short term.)


Comments on this page:

To take another step back, the entire reason CM systems exist breaks down into two things:

  1. Applying configuration to multiple systems
  2. Providing a higher level, possibly more sensible way to define a system configuration

The first problem is caused because package management is inadequate. If creating a site-specific config package and pushing it out was easy, we wouldn't need 90% of what most CM systems provide.

The second problem is caused because every program developer historically felt the need to invent a new config file format. Heck, the Windows Registry's hierarchy and limited set of data types looks like a panacea of order compared to most Unix systems /etc directory. I don't see a way out of this unless someone was to either replaces daemons with newer versions, or does a "sane config" fork of most standard system daemons that forces the config files to use YAML/JSON/XML.

To me, Chris, you're just looking for an excuse for not using CFEngine or Puppet, but you need to honestly talk with yourself about whether this is true or not.

About dying some of the tools from the market, sure, I hope too the landscape will shrink a bit. But I already see how this works for me. I am dedicated to cfengine 2.x, which is end of life and which can be seen as a separate (similar, but separate) tool from CFEngine 3.x. It won't take too long to migrate, and the hardest part, learning new paradigm, took nothing -- I already learned it with cfengine 2.x.

By rjc at 2013-09-27 04:30:47:

@zdw

I agree with your two points, there are of course other ones, like:

  1. Automation (it's not the same as your point 1.) - changes made server(s) propagate automatically to all nodes - no need to push them individually (or en masse) manually.
  2. Heterogenous environments:
    • different hardware platforms
    • different operating systems
    • different OS releases
    • different distributions in terms of Linux
    • different software version
    • different roles:
      • server
      • workstation
      • laptop (or other mobile device) - push model fails here.
  3. Not just higher level config but a higher level (pseudo) language which aids in decision making - Shell scripts are fine but what will you do if you have to support Windows? Choose YET ANOTHER scripting language ;^)

These are only the ones from the top of my head and I've probably missed a couple.

What I don't agree with is your comment:

The first problem is caused because package management is inadequate. If creating a site-specific config package and pushing it out was easy, we wouldn't need 90% of what most CM systems provide.

I agree with the first sentence. Not so much with the second. Look at point 2. - even if creating such a package and distributing it WAS easy, this simply wouldn't work if you have to support OS X, Linux and Windows. Heck, even with Linux if you have to support a couple of subsequent Ubuntu LTS releases, and maybe just one other distro, sure you can build three packages... but rebuild them (as in, make a binary package) every time you want to make a change and then ask your package manager to install it!? Then add: two different hardware platforms (x86, amd64), one other OS (OS X 10.8-10.6)... That's insane! Point 3. aids you with that.

Regards,

rjc

@rjc

Given that modern package managers are exceedingly capable of managing dependencies, and that config files generally don't change radically between minor versions of a software package, why not put the config files in a separate package? A config package would change very rarely, relative to binary rebuilds for patches/security fixes, and could be rolled into whatever deployment/system image creation system that's being used.

This radically simplifies the number of combinations issue you brought up, which definitely is an issue given the current bundled binary/config model. I wouldn't be surprised if all the platforms mentioned could share identical versions of simple config files like resolv.conf or ntp.conf, just packaged up into their respective package formats in an automated fashion.

There's already ample precedence for splitting software into development, documentation and library packages, so a separate config package isn't a radical new idea.

I'd rather have a CM system that runs server side, performing all the logic of creating config files and packages and working with the system imaging/packaging/update automation bits, than the "run an agent that requires a bunch of dependencies on every box and acts like a stupid robotic sysadmin" model which seems to unfortunately be ascendent.

"and I kind of hope that not all of them survive, because if they all do that's a lot of duplicate or near-duplicate effort"

We're talking about a world that has how many mainstream Linux distributions, each of which (on the desktop) tries to push one or more of several mainstream desktop managers?

Even in BSDland, you've got your choice of Free, Net, Open, or Dragonfly, although I suspect that the latter is confined mostly to desktops.

By rjc at 2013-09-27 13:51:09:

@zdw

The below example is obviously for puppet - now, it's not a strong preference but it is what I am using at the moment. It has a high enough abstraction layer that I don't have to play with Shell, Perl, batch, etc.

A VERY short and quick example:

class hosts {
  Host { ensure => present }
  $nodes = {
      'one'   => { 'ip' => '192.168.0.1' },
      'two'   => { 'ip' => '192.168.0.2' },
      'three' => { 'ip' => '192.168.0.3' },
      'four'  => { 'ip' => '192.168.0.4' },
      'five'  => { 'ip' => '192.168.0.5' },
   }

   create_resources(host, $nodes)
}

Creating a package especially for five lines of text? Add another Linux distro, OS X and Windows while we're at it. Let's assume I do that. Now, I know a bit about DEB packaging, virtually nothing about RPM, PKG or Homebrew for OS X... what about Windows? With a sane CM I don't have to worry about any of it!

You can think of CFEngine and Mark whatever you like but there's a lot to be said about the idea of convergent maintenance.

With package manager behind our CM, what if local admin removes one or two of those entries? What then? Lock the package manager just to handle configuration because we need to reinstall a package every 15 minutes? What about OS/software updates or new installs - we need to make sure we hit the time window... those pesky lock files ;^)

package management != configuration management

and, IMVHO, should not be treated as such.

You've mentioned that you can simply build a package with your preferred config. That's so '00s ;^) With all seriousness though, apart from the probelm described above, what if you have multiple machine types withing different groups which require different setup for the same kind of software? Again, build a package for each? Then you'll still need to use some logic to apply different config packages to different groups of machines. Now you scratch your head and reinvent the wheel. Why would you do that to yourself!?

Home grown CM systems based on package management might as well be good enough, but only for small and/or homogeneous environments without under 30-minute policy compliance. That's also assuming there's more than one of you who knows how the system works, you're institutionalised and never going to change jobs or documentation is immaculate in case you take a week of sick or have an accident on the way to work. Even if all of that is the case, your company should seriously look into how they invest their money - creating, effectively, yet another CM system is not necessarily time well spent in terms of PD or a way to make yourself indispensable.

This radically simplifies the number of combinations issue you brought up, which definitely is an issue given the current bundled binary/config model. I wouldn't be surprised if all the platforms mentioned could share identical versions of simple config files like resolv.conf or ntp.conf, just packaged up into their respective package formats in an automated fashion.

With only two distinct sites I'll need two different DNS resolver and NTP configurations - with ONLY two Linux distros (DEB and RPM for the sake of argument) that's 8, EIGHT! (sic!) individual packages ( 2 sites * 2 distros * 2 config files ). Again, with a sane CM I only need two "recipes". And that's without OS X ;^)

I'd rather have a CM system that runs server side, performing all the logic of creating config files and packages and working with the system imaging/packaging/update automation bits, than the "run an agent that requires a bunch of dependencies on every box and acts like a stupid robotic sysadmin" model which seems to unfortunately be ascendent.

Well, they ~~all~ run server side. I take it that you meant agent-less push model. Let's ignore all of the above problems for now. Your example is only going to work if you have:

  • direct access to the machines
  • they're always on

It will not work for machines which:

  • are frequently off-site (i.e. laptops)
  • aren't always on (i.e. not a server)
  • OS environment isn't heterogeneous (i.e. client being a different OS than server)

With a push model you have to run an "agent" anyway, be it an SSH server - again, I hadn't mentioned Windows here ;^). Most agents nowadays rely on software already present in the system, come as bundles or are self-contained - no need to agonise over dependencies.

Still convinced that CM based on package management is the way to go?

Regards,

rjc

@rjc

You last set of arguments boils down to roughly:

  1. package management != configuration management
  2. Admin's will change things and you won't know that it happened
  3. Making lots of packages that work on all platforms is hard
  4. Some systems require config to be in their own weird format (Windows), so we need a cross platform way of expressing configuration and getting it
  5. Non-server systems like laptops and machines that aren't always on or move around the network are a challenge

Regarding #1, I'd argue that package management should be a superset of config management.

Regarding #2, Most packages contain checksums for each file (or could if they don't) and there are tools like tripwire that do similar things. These tools would cover far more of the system and have security benefits over the limited, pinpoint file checks most CM systems do.

Regarding #3, that's a tooling issue. I'll be the first to admit that most if not all package management systems could be greatly improved. Imagine what would happen if friction of package creation is minimized or gone. You log into a system, change a config file then run a "package create foobar-config" and out pops a datestamped/checksummed package that you can install on other systems with the same OS. Imagine tools where you could open a config package for platform X on platform Y, then roll it back up and deploy it. As long as you're just changing config files and you don't need a full compiler/toolchain, a lot of the cross platform package generation problems are no longer issues.

Regarding #4, I totally agree with you. Looking beyond Windows at legacy systems or devices like switches, routers, UPS's, and other infrastructure pieces, I agree that there needs to be some sort of system for applying changes in the appropriate manner. I'm just not convinced that systems with package management should be managed the same way as those "dumb" devices.

Regarding #5, that's an implementation detail that could be overcome in myriad ways, and isn't uniquely solved by CM system. A simple way to kick off new package installation (say a web server with a signed "update yourself" file), coupled with a logging infrastructure that returns errors or other system data would do just as good of a job.

I do welcome the discussion on this, as I hope you do.

By cks at 2013-09-27 17:44:27:

Although I used to feel somewhat otherwise (and once wrote a whole series of entries in favour of it), I've come around to believing that doing configuration management through system packages is the wrong approach for both pragmatic and high-level reasons. A proper explanation of this view requires an entry; the short version is that pragmatically it clearly doesn't work well today and at a high level I think it's working in an awkwardly indirect way and at the wrong level of abstraction.

Written on 27 September 2013.
« Trying to explain my harshness on configuration management tools
Why I put configuration management systems over packaging systems »

Page tools: View Source, View Normal, Add Comment.
Search:
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri Sep 27 00:41:20 2013
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.