Another problem with doing your own sysadmin automation

May 11, 2008

In addition to the stuff in AutomationCosts, there's another problem (or cost) with custom sysadmin tools: they're almost always at least not-good software, and sometimes they're outright bad. They pretty much can't help but be because of how they're created.

The typical custom tool is written by at most a small group of people (who are often not actively practicing programmers), is not looked over by very many people, and on top of that is often written in a bad language for programming. I'm not saying that you have to be an active programmer to write good programs, but I think that it helps; the odds are stacked against a small team that doesn't do much programming producing something good. Similarly, having lots of people poke at your work helps improve its quality, especially if they are outside people.

(Yes, there are sysadmins who are good, active programmers, but generally the people who really like programming become programmers, not sysadmins.)

In short, your custom tools just don't get the degree of attention and involvement that vendor software or open source software gets, which stacks the odds against them being as good as such software.

The usual rejoinder to this depressing view is that you'll generalize your custom tools and then share them for the world to adopt and improve, which will get you all the benefits of open source software in general. Unfortunately this does not seem to work in practice; the LISA conference proceedings are littered with tools that have gone basically nowhere since being announced. You might be an exception, but history suggests that the odds are still stacked against you.

Comments on this page:

From at 2008-05-11 09:17:35:

Consider though, the advantages to custom written sysadmin tools. They do exactly what we want, when we want it.

I've yet to see a commercial tool do much more than 80% of what I want.


From at 2008-06-02 16:40:26:

I might be the special case that proves the rule: the frustrated programmer who's working as a sysadmin.

Everywhere I got a programmer job, the local sysadmins were pretty bad, and so people would come to me to fix things, repair infrastructure, show them how to use Microsoft Outlook... eventually I threw in the towel and started applying for syadmin jobs. Actually easier to get jobs, since I don't have to show a certificate in Java development from "no no, we don't recognize that institute, only the OTHER institute" and more years experience than the language has been in existence.

I find myself automating things for two major reasons:

  • the developer team built something that is making my pager go off. I could either phone them, and impotently try to cajole them into fixing it, or I can fix it myself, document the fix, and check it into their code repository so that I won't get paged for it ever again.
  • management has thrust upon me some tool they read about in a magazine. It cost $$$, and it's meant to solve EVERY problem in a domain (makes for a wider market niche for sale), but it doesn't solve ANY problem well, and may not even fit our own needs. I find myself either writing code to fill in the cracks, or to provide a GUI interface to a synthesis of tools (some $$$, some free and open) that do cover all the company's needs.

Case in point: Last week, I could've spent four days trying to fix a web-based file sharing tool, legacy product purchased $$ by the last sysadmin, but instead I spent four hours writing something that was feature-for-feature compatible, and used apache's mod_auth so it could be tied into LDAP or RADIUS someday.

Counterexample: We used to spent about four manhours a week tuning our spam filters with spamassassin. Now we buy spam-filtering from a third party, and get the same success rate. Cheap at twice the price.

Some automation is expensive, but not all.

 --Mozai, moc.iazom@sesom
Written on 11 May 2008.
« An advantage of interpreted languages
The history of readdir() »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun May 11 00:10:18 2008
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.