My view on what will kill 'traditional' system administration

February 5, 2012

Phil Hollenback recently wrote DevOps Is Here Whether You Like It Or Not, in which he writes that traditional system administration is dying. While I sort of agree with him about the death, I don't think it's necessarily for the reasons that Phil points to.

Fundamentally, there has always been a divide between small systems and large systems. Large systems have had to automate and when that automation involved applications, it involved the developers; small systems did not have to automate, and often do not automate because the costs of automation are larger than the costs of doing everything by hand. Moving to virtualization doesn't change that (at least for my sort of system administration, which has always had very little to do with shoving actual physical hardware around); if you have only a few virtualized servers and services, you can perfectly well keep running them by hand and it will probably be easier than learning Chef, Puppet, or CFEngine and then setting up an install.

(If you're future-proofing your career you want to learn Chef or Puppet anyways, so go ahead and use them even in a small environment.)

There are two things that I think will change that, and Phil points to one of them. Heroku is not just a virtualization provider; they are what I'll call a deployment provider, where if you write your application to their API you can simply push it to them without having to configure servers directly. We've seen deployment providers before (eg Google App Engine), but what distinguishes Heroku is how unconstrained and garden variety your API choices are. You don't need to write to special APIs to build a Heroku-ready application; in many cases, if you build an application in a sensible way it's automatically Heroku-ready. This is very enticing to developers because (among other things) it avoids lockin; if Heroku sucks for you, you can easily take your application elsewhere.

(This has historically not been true of other deployment providers, which makes writing things to, say, the Google AppEngine API a very big decision that you have to commit to very early on.)

Deployment providers like Heroku remove traditional system administration entirely. There's no systems or services to configure, and the developers are deeply involved in deployment because a non-developer can't really take a random application and deploy it for the developers. If there is an operations group, it's one that worries about higher level issues such as production environment performance and how to control the movement of code from development to production.

The other thing is general work to reduce the amount of knowledge you need to set up a Chef or Puppet-based environment with certain canned configurations. Right now my impression is that we're still at the stage where someone with experience has to write the initial recipe to configure all N of your servers correctly, and you might as well call that person a sysadmin (ie, they understand Apache config files, package installation on Ubuntu, etc). However it's quite possible that this is going to change over time to the point where we'll see programs shipped with Chef or Puppet recipes to install them in standard setups. At that point you won't need any special knowledge to go from, say, writing a Django-based application to installing it on the virtualization environment of your choice. This really will be the end of developers needing conventional sysadmins in order to get stuff done.

The general issue of the amount of hardware in a small business (and virtualizing the hardware) ties into a larger question of how much hardware the business of the future is going to need or want, but that's a different entry. I will just observe that the amount of servers that you need for a given amount of functionality has been steadily shrinking for years.

Sidebar: what virtualization does change now

I think that plain virtualization does mark a sea change today in one way: it moves sysadmins away from a model of upgrading OSes to a model of recreating their customizations on top of a new version of the OS. Possibly it moves away from upgrading software versions in general to 'build new install with new software versions from scratch, then configure'.

This is partly because the common virtualization model is 'provide base OS version X image, you customize from there' and partly because most virtualization makes it easy to build new server instances. It's much easier to start a new Ubuntu 12.04 image than it is to find a spare server to use as your 12.04 version of whatever.

(Note that virtualization may not make it any easier to replace your Ubuntu 10.04 server with a new 12.04 server; there are a host of low level practical issues that you can still run into unless you already have a sophisticated management environment built up.)

I don't think that this is a huge change for system administration, partly because this is pretty how much we've been doing things here for years. We basically never upgrade servers in place; we always build new servers from scratch. Among other things, it's much cleaner and more reproduceable that way.

Comments on this page:

From at 2012-02-09 06:52:10:


1. give developers root rights in all the hosts in your network; 2. wait until they seriously mess everything up; 3. profit.

Even in the 'best' gui configuration management tools like Microsof SMS server (or whatever it is called nowadays - it is a really nifty gui tool), if the one using it does not have a clue how stuff works, they will mess everything up.

In a network you need someone who knows how netbooting works, how ldap/kerberos/permissions/drivers/mount points/switches/routers/... work. That someone is the sysadmin.

Developpers are not in the least interested in this stuff, they just want to worry about their application, and if the app works out of the box as root/administrator, hey, give everyone that right. It works for them, so that is solved.

We know by know how that way of thinking has (not) worked well for the reputation of MS in the past.

So devops: yes. Sysadmins who code are devops.

Developpers are no admins and they will never be.

And, by the way, in a virtual enviroment I have rather have the servers do stuff and not just waste their resources with that pig that is Puppet (and I think the people paying for the servers would rather have their servers do something useful for their companies and not only distributing and managing their software). Chef being ruby as well it should not be very different but I cannot compare it. Anyway, why should I need ruby if we do not do any ruby?

For a usenix comparison about Cfengine vs Puppet just read and cry if you are a puppet user (or laugh if you are a cfengine user).

Written on 05 February 2012.
« Link: Filenames.WTF
The advantage of HDMI for dual displays »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun Feb 5 23:56:02 2012
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.