Wandering Thoughts archives

A bunch of my sysadmin work seems to be like gardening

Especially, it seems like weeding, or at least how I've read weeding described (I'm not a gardener myself so I have limited experience here). By that I mean that it's plodding and often boring, and involves going through our systems to trim back or pull out entirely various things that were once necessary (well, probably) but that have been neglected and are now at least overgrown and perhaps outright bad. It's painstaking work because I have to make sure that what I'm pulling out isn't going to yank up anything important with it (or suddenly turn out to not be a weed after all, but something important we still need). And there's always more of it in long-lived systems. They accumulate clutter and then the clutter gets overgrown as people's memories about why it's there and what makes it necessary and so on fade away.

(The other part of this is that once you have a workaround for a problem, you usually don't carefully test each new system and OS version and so on to see if the problem is still there, and keep in your documentation a current matrix of what needs it and what doesn't. Once a workaround is in there you often stop thinking about it, which makes it easy for a workaround to become unnecessary.)

Thinking of this sort of work as gardening and weeding makes it a little bit easier for me to keep my enthusiasm up for carefully grinding through a set of tests to verify that probably something can be pulled out. It's just like pulling the weed of your choice; you do it and do it and keep doing it and there's another one hiding in the corner and so on. You have to accept that it's a process and just keep going, bit by bit. And in system administration, some day we can hope to turn around and have this particular weed be all gone.

Sidebar: the weed I'm pulling that inspired this thought

We have a long-standing carefully enforced limitation in our local system to propagate passwords that /etc/group lines can be no more than 512 characters long. Since we have several groups with memberships that normally exceed this, we have some annoying workarounds for this limit. If a 512-byte limit is no longer necessary, we'd like to get rid of it to make our lives simpler.

But, well, this is a lot of careful work to check that nothing is going to break. It's not just programs that directly use /etc/group through the libc routines, either; we have some awk code (somewhere) that processes /etc/group to generate data files for Apache's basic authentication. That probably won't break, and Apache can probably deal with big groups, but I'd better check it all, hadn't I. And go looking for other scripts that process /etc/group to do something, too, of course.