How I (once) did change management with scripts
When I read Philip Hollenback's latest entry and it mentioned someone doing (change/system) management through shell scripts (instead of, say, Puppet), my first thought was 'hey, I've done that'. So I might as well write up how I did it, either for someone to use or in case people want to marvel at the crazy person.
(Now, a disclaimer: by now this was more than half a decade ago, and some of my memories of the fine details have undoubtedly faded (ie, are now wrong).)
The basic environment this happened in was a lab environment with (at its height) on the order of a hundred essentially identical PC machines running Linux (this is the same environment where we needed efficient update distribution). Most of the system management was handled through packages and automatic package updates, but every so often there was something that was best handled in a shell script.
Each separate change was a separate little shell script, all of which
lived in a common directory (actually one directory for each OS
release). Script filenames started with a sequence number (eg they had
names like '
01-fix-something'), and scripts were run in sequence. The
driver system kept track of which scripts had already succeeded and
did not re-run them; a script that exited with a failed status would
be retried the next time the driver system ran. The driver system ran
once a day or (I believe) immediately after system boot, and processed
scripts after applying package updates. Scripts were expected to check
if they were applicable before doing anything and exit if they weren't
(with status 0 if they were definitely not applicable to this system or
with status 1 if they should be retried the next time).
(If I was doing this again I think I would make the driver script not run further scripts if an earlier one failed. In our case all of the scripts were basically independent, so it didn't matter.)
There was no mechanism to rerun a script if it changed; if I changed a script and wanted to have it rerun, I needed to give it a new sequence number. If a script became unnecessary for some reason, it was just removed.
All of this is actually quite short and simple to implement, and it worked quite well within its modest goals. It was not particularly difficult to write scripts, they were automatically executed for you, all machines were kept in sync, and a newly (re)installed machine would automatically pick up all of the current customizations. These days, you would put the entire directory of scripts into a VCS (and you might distribute it by having the workstations check out a copy from the central repo).