Operators and system programmers: a bit of System Administrator history

October 17, 2012

I saw yet another meditation on the difference (if any) between operations and system administration make the Twitter rounds recently, which has finally pushed me over the edge to say something about this. I want to talk about the history behind this apparent division, at least as I see it from my perspective.

(My disclaimer here is that I was not around at the start of this story, only towards the end and only in an academic environment. So my perspective may be skewed.)

A long time ago, or at least back in the early 1970s, most computers were big and expensive and also required near-constant tending and physical work. If the computer was running, someone was forever shuffling punched card decks around, mounting and unmounting tapes and disk packs, collecting line printer printouts and putting new paper into the printers, and often doing manual steps to prepare the mainframe's OS for the next job (or clear out the previous one, or both). The people doing all of this were were (mainframe) operators, and for obvious reasons this was considered a relatively low-skill, low-prestige job. Mostly you needed to be able to follow instructions in big procedures manuals.

(This is a bit of a stereotype.)

At the same time there were also system programmers, because the mainframe was so expensive that it was worth it to pay expensive programmers to keep it running smoothly. System programmers generally worked on writing and fixing OS-level components and utilities, say components of your job control and batch management system; where application programmers might work in COBOL, system programmers might work in System/360 assembler. Being a system programmer was a high prestige job and was considered more challenging than being just an application programmer. A Geoff Collyer quote from a later era captures the feel rather well:

This is what separates us system programmers from the application programmers: we can ruin an entire machine and then recover it, they can only ruin their own files and then get someone else to restore them.

System programmers also had the job of actually installing and setting up mainframe operating systems, applying vendor bug fixes, and so on, because back in those days none of this was what you could call 'user friendly' or 'easy'.

The original Unix systems were not big mainframes but they didn't exactly come with detailed, easy to follow procedures manuals; my strong impression is that setting up and operating early Unix systems pretty much required a programmer. As a result most or all of the people who ran early Unix systems were effectively system programmers out of necessity. If you ran a sufficiently big and important Unix system you might have some 'operator' helpers to do physical work like changing tapes, especially if you got to drop your Unix machine into a machine room that already had operators to tend to its mainframes, but the core people who kept the machine running were all system programmers.

As Unix systems matured a bit, increasing amounts of the work necessary to keep them running didn't need actual system programming, just (dangerous) root permissions and some judgement; this was things like adding and removing users, kicking printer queues around, and so on. Places that already had operators might have their (Unix) system programmers create user-friendly menu systems or the like so that the operators could safely have this work delegated to them, but places without operators started hiring junior people to handle this. Not infrequently these junior people were some sort of programmers and were expected to upskill themselves into full system programmers over time. As Unix systems became more and more mature, running one in routine situations had less and less need of system programmers instead of these 'system administrators' (for example, pretty soon you could install and run a Unix system without having to understand the insides of configuring and compiling a custom kernel for your specific hardware). That this happened is unsurprising; it's basically the standard trajectory of a field, as what was new and unknown becomes routine and then mechanized.

(In addition Unix systems also had their 'operator' level work to be done. No matter what particular operating system you're running on what hardware, someone has to change the printer paper. This level of work on a Unix system was no more prestigious or better paid that being a mainframe operator, and for much the same reasons. Still, you could aspire to move up.)

During this slow change, system programmers and a system programmer mindset hung around many old and well-established Unix shops for a very long time (and some vestiges of it linger even today in places). This was the era when it was not surprising for a 'senior system administrator' to be able to write serious system programs like a mailer, a Usenet news server, a nameserver, or a language, and it was kind of expected that every senior sysadmin worth the name was really a system programmer; you can see this fairly vividly in LISA proceedings, especially older ones. By extension this attitude somewhat pervaded the general Unix system administration culture of the time (and it didn't hurt that scripting is genuinely useful on Unix).

What we are left with today is a confusing blend of a situation. A significant amount of the cultural background of Unix system administration comes from system programming, while the actual work required to keep machines running spans a vast gamut. But we often expect that one person or a small team will cover the whole gamut and for historical reasons we default to applying the label 'system administration' to the whole result. All of this makes it hard to break things up when we want to talk about how to do things in a large scale environment, one where we will have people with different skill levels who will work on significantly different things.

(In short, no one agrees on what to call things and 'system administration' is too ambiguous and too broad.)

This has wound up being rather more thinking aloud than I was expecting when I started writing.

Sidebar: why mainframes were different

My view is that mainframes were always big enough (and expensive enough) that you needed a decent sized organization to run one. Mini and microcomputers have been blessed with much smaller sizes and much lower costs, so you could acquire one and tell a single person or a very small group that they were in charge of everything to do with it.

(In the early days of Unix this was often 'whichever grad student was closest and sufficiently slow moving'. To be honest, it's still that way around here every so often.)

Sidebar: a little bit of local trivial

As you might expect the university has a long history on the system programmer side of things, a history that still lingers in various ways. One of them is that most system administrators here were officially 'System Software Programmers', a job classification that was considered superior to (and better paid than) mere 'Application Programmers'.

(I'm out of touch with official job classification fun here, so this may still be true.)

Comments on this page:

From at 2012-10-18 12:45:19:

Thank you for the background on this. I had not realised that Systems Programmers were a thing. Recently, I have run across "want ads" for Systems Administrator positions where they want an old school Systems Programmer, not a new variety SysAd.

I wonder what skills, and to what degree, make up a modern successful SysAd?

Written on 17 October 2012.
« Switch flow control and buffering: what we think was wrong in our iSCSI network
A danger of default values for function arguments (in illustrated form) »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Wed Oct 17 02:22:10 2012
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.