One reason that I call us a midsized environment

November 3, 2010

I mentioned earlier that I have a number of reasons to call us a midsized environment. One of them is how we need to run our environment, in that we are in the middle between two extremes of how to deal with systems.

On the one hand, we are sufficiently large that you don't want to do things by hand and it makes sense to automate at least some things. We've long since grown past the point where all of our machines could be set up separately or run individually (and not just because NFS requires synchronized UIDs), which is the practice that you often see in small environments.

On the other hand, we are not so large that we have to automate things or die. Once you reach a certain size, it is basically impossible to run your environment in any way that requires routine hands-on attention to individual machines; you must automate every such thing or you'll be unable to keep your environment up (or at least unable to improve it at all). This is the size where people get fanatical about automated deployments, automated management with Puppet or Cfengine, automated monitoring with your choice of tool, and so on.

We sit in the middle of all of this; we have some automation but also some things that we do by hand, partly because we like it that way and partly because it's too much of a hassle to build and maintain an automated system. Hence, mid-sized.

(I maintain that sitting in the middle like this is sensible in at least some cases, and there are good reasons not to immediately jump up to the Jumpstart/Kickstart, Puppet, etc etc world. Part of it is the costs of automation, but the full discussion deserves another entry.)

(See also this older entry on levels of automation.)


Comments on this page:

From 77.249.14.105 at 2010-11-03 15:05:16:

we are midsized too, but without management systems it would simple not be possible to keep stuff working like it is now. In our case we use CFengine (version 2 still) and for the Windows side wpkg. Sure, it takes some time to get used to using the tools, but now I can spin a whole lot of new machines knowing they will be exactly as I want and they will remain like that. No entropy for us, self healing systems galore.

We have gone from putting out fires and mopping the floors to doing project work and I can tell you: huge difference.

And you do not need to write your own system, there is enough choice and choice is good.

-- natxo

From 139.78.143.201 at 2010-11-03 17:32:26:

Yeah, ditto. You more or less know what I run, Chris, and we use cfengine and kickstart -- but I stop short of doing everything on virtual machines, which I think you do more of. So I think mid-size means "picks and chooses automation instead of being forced to use all you can" perhaps. :)

Random

From 71.116.163.6 at 2010-11-04 11:53:06:

Our environment is definitely smaller than yours, but I've been using Kickstart to build machines since I started back in 2001. In part that was because I inherited a big mess, including a nasty imaging-based system for setting up workstations that (1) required a bunch of manual tweaking to get working once the image was installed and (2) meant that the machines were starting off needing a whole slew of updates just to get up to the current package set.

I ended up writing a set of scripts that would pull information out of a database (a file with Perl data structures at first; now YAML) and populate templates for Kickstarting, generate the PXE boot file, and so on. Worked great, and, over time, I expanded the system so that it also installed and ran a script that did most of the machine-specific tweaks -- configuration files, but also installing different package sets depending on the role the machine had been assigned.

I still don't have Puppet or another configuration-management system set up, but I want to. In the meantime I can mostly get away with ssh loops, and for significant changes to workstations I can just tweak my scripts or templates and reinstall them by rebooting. Knowing that I can get back to a known good state has made my life much easier, and the network install infrastructure has also made it possible for me to replace a couple of servers on fairly short notice -- with configurations and data pulled from backups, of course, but at least the basics were taken care of quickly letting me worry about the important bits instead of the basic install.

  -- Claire
Written on 03 November 2010.
« Redirecting from HTTP to HTTPS is a bad security idea
What we (currently) use virtualization for »

Page tools: View Source, View Normal.
Search:
Login: Password:

Last modified: Wed Nov 3 01:16:17 2010
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.