Wandering Thoughts archives

Why I hate $LANG and locales on Unix

; cat foo
be+c 1
be+f 2
bed  3
; look bed foo
bed  3
; sort -o foo foo
; look bed foo || echo failed
failed
; cat foo
be+c 1
bed  3
be+f 2
; echo $LANG
en_US.UTF-8

Locales are intrinsically a user interface issue; you want to present information to the user in their specific local format. The Unix $LANG approach is intrinsically because commands have no idea whether they are presenting information to the user, or to other commands; either way they choose, they cannot win.

But the ways that things lose are different in each option. If programs ignore the locale, they present information to the user in a somewhat less desirable format. But the current locale approach actually breaks things on Unix, as this example neatly shows.

The locale approach is superficially attractive but deeply harmful for Unix systems, because it does fundamental damage to the idea that programs can be used as internal building blocks in bigger things. There once was a day when sort was a useful component; as demonstrated, that day is now effectively over.

Sidebar: how to work around this

The GNU sort documentation has scary warnings, so it appears that you need to set both LANG and LC_ALL to 'C', just to be sure. (It looks like you can't leave them unset unless you unset all of the LC_* environment variables, but setting these overrides the others.)

How not to set up your DNS (part 12)

Last week Securityfocus, home of a number of reasonably active mailing lists that our users like getting, changed their nameservers from the Symantec corporate nameservers to UltraDNS. And then the Symantec corporate nameservers promptly stopped giving useful answers to queries about the securityfocus.com domain (instead returning the standard 'I have nothing to do with them' results of a referral to the root zone).

Since the mailing lists are reasonably active and we do check to make sure that the MAIL FROM domain actually exists (much like most of the rest of the Internet), our nameserver had the old NS records cached. The ones that now disclaimed authority, thereby creating temporary DNS failures when we tried to verify the MAIL FROM of new email from Securityfocus mailing lists.

(We eventually 'solved' the problem by restarting our caching nameserver, thereby purging its cache of the old NS records.)

All of this points out the golden rule of DNS server transitions: you need both the new and the old NS servers answering queries (and with the same answers). In a sense it is a form of garbage collection; you shouldn't turn off the old DNS servers while anyone can legally still have a cached reference to them.

Two approaches to Unix environments

There are two general approaches to coping with the various environments of Unix accounts that I've seen people adopt. The first is taking the default environment that each system comes with (perhaps with minimal and easy customization for stuff that really bugs you); the second is building a completely customized personal environment that you then drag everywhere.

(To stereotype, people who follow the first sort feel that life is too short to spend futzing with your environment when the one that someone else worries about works well enough, while the other side similarly feels that life is too short to live without something that fits your desires to a T. Note that this isn't to say that the first sort doesn't care about the environment they use, just that what they care about is probably indirect issues, like a more generally consistent GUI, instead of specific features that the second sort are solidly locked to.)

The split influences a lot of attitudes in relatively deep ways. For example, the first sort if much more likely to really like moving to Apples; for the second sort, the Apple GUI is at best neutral and often negative, since it is not their environment.

I am a very strong second sort of person, and I am currently wrestling with the true nemesis of my calling: moving my personal environment to a new system (in this case Fedora Core 5 on an x86_64). Because I have so strong particular opinions on things, even little changes are completely irritating; this makes dealing with things like font issues rather interesting.

The other annoying bit of moving my environment is trying to figure out how to make it do various useful things normally provided by the system environment. For example, FC5 will conveniently automount USB keys and DVDs and so on, if you use GNOME or KDE; now I get to figure out all the magic bits and daemons so I can add them to my own environment.

Oh well, at least I'll come out of this experience better informed about the inner workings of some of the black magic involved.

Sidebar: why font issues bug me and matter

Every OS upgrade seems to futz around with fonts so that things come out wrong for something. And don't get me started about Xft, apart from saying that there sure don't seem to be any good monospaced fonts for people who like their inter-line spacing relatively narrow.

Why do I care about fonts so much? Because the layout and spacing of a great many things on my screen, and how I organize space, is strongly tied to how much space things like 80x24 xterm windows take. Change the fonts, those sizes change, things don't fit together any more, and I grind my teeth in your general direction.

(Don't ask what I feel about the possibility of changing display resolutions.)

Some wise words from Henry Spencer on backups

Henry Spencer recently wrote some very useful words of advice on backups on a local sysadmin mailing list. They struck me as the sort of things that are useful enough to share more widely, so with Henry's permission I'm putting his message here. (I thought about running just part of his email, but the more I read it, the more I wanted people to see all of it, so I'm just going to put up the whole thing.)

So, in Henry Spencer's own words:

...So please don't be put off doing a simple thing that will produce significant benefit in most cases, such as storing backups in the next building, just because there exist some "movie plot" scenarios in which this would not be good enough.

I concur. (And I speak as one of the few people on this list who's been running machines on campus long enough to remember the Sandford Fleming fire.) Remember also two things:

(1) A disaster big enough to wipe out both your building and the next building over is likely to have repercussions severe enough to make the up-to-dateness of your offsite backups somewhat secondary.

(2) A wonderful offsite-backup plan which is so inconvenient that it is followed only fitfully is worse than none at all.

There is something to be said for doing an occasional very-offsite backup. But for the weeklies and monthlies, above all you want a plan which is practical enough and convenient enough that you will FOLLOW IT consistently, month after month after month. Hauling a pile of media to and from a remote location gets tedious quickly.

Bear in mind, too, that by a corollary of Murphy's Law, the time when a backup will be most needed will be when the relevant sysadmin is out of town. You want an offsite-backup location that your assistant (etc.) can get access to when necessary; the top shelf of your hall closet is out. If your offsite backups are stored in the next building by informal arrangement between you and the sysadmin there, make sure that other people in both places know about it. You may want to have a formal authorizing letter ("Joe Blow and his staff from Dept. XYZ are authorized to remove or exchange the tapes on the bottom shelf of storage cabinet 3 at any time") on file in case everybody technical at the far end is away.

The one halfway-plausible accident that just might manage to affect two adjacent buildings is a fire. Not because the fire is likely to spread to the second building, but because water and smoke don't necessarily respect building boundaries. (When Sandford Fleming burned down, the firemen spent six hours pouring water in from all sides... and at least one adjacent building was closed due to flooding; indeed, there was flooding as far away as Queen's Park subway station.) Smoke in particular can get into places you'd never think it would reach -- closed drawers, etc. -- and the soot it leaves can be quite corrosive.

There is one simple step you can take that will make your offsite backups much less vulnerable to such indirect hazards: bag them in airtight zip-lock bags. In fact, this is worth doing for the most recent set of on-site backups too -- a serious fire anywhere in your building can expose your computing facility to water and smoke even if the fire never gets anywhere near it.

The hazards of smoke and soot are something I hadn't previously thought of, and the zip-lock bag trick strikes me as both very clever and nicely simple. (I have a weakness for simple, low-tech solutions to problems.)

(PS: for University of Toronto people who stumble over this entry and want to be on the local sysadmins mailing list, you can get on by sending email to ut-admins-request at the domain utcc.utoronto.ca.)

Why writing documentation is no fun

I recently had an insight about why so many sysadmins (and programmers) would rather have a root canal than write documentation for any length of time. (While some of these feelings can be blamed on people feeling unskilled, not all of them can. Certainly I can't claim with a straight face to be completely unskilled, and the idea of having to write documentation for any length of time still makes me gloomy.)

I think it's because documentation doesn't do anything on its own, leading to a lack of feedback on your work.

Us sysadmins and programmers are used to doing things where we get immediate feedback. When you build a system or write code, you can see your work actually doing things, right then and there. (Sometimes what you'll see is your work blowing up, but you are still getting feedback.)

But when you write documentation it just sits there until someone else reads it. (And you can't really do anything with incomplete documentation, unlike incomplete code or systems, where you can at least exercise what you have.)

So, in short: seeing your stuff doing things is fun. Look, you did that, isn't it neat? Documentation doesn't do things; it just sits there, with none of the usual fun to be seen.

Plus, there's another aspect of this: sysadmins and programmers work in an environment of fast feedback, so I think we get habituated to feel that getting that feedback means you are working productively, and not getting that feedback means that something is wrong. So writing documentation means you're not getting that feedback, which means that something in the back of your mind feels that you're not getting things done, even if you're turning out the words.

(Matters become even more demotivational if your co-workers don't give you much feedback; there is the pernicious feeling that what you've worked on is clearly valueless. No one likes to work on valueless things, even if they are technically well executed.)

Why Postfix is not my favorite mailer

Today I was unpleasantly surprised to discover that Postfix defaults to having a 50 megabyte size limit on user mailboxes, after which email bounces. This default limit is apparently sufficiently unimportant to not be mentioned in the 'basic' or 'standard' general configuration documentation.

Perhaps this will strike people as a petty reason for disliking Postfix; after all, it's just one configuration option. But it is the symptom of a deeper problem.

Since Postfix people have made one dangerous random arbitrary default (and not drawn attention to it), I can't count on them not to have made more that I just haven't tripped over yet. In other words, I can no longer count on them to have a sane default configuration.

This means that I have to examine the default for every configuration option Postfix has, just in case it's going to eat email in the future. If you have never looked, Postfix has a lot of configuration options; the HTML page documenting them all is over 400 Kb. Many of them will be unimportant, many of them will have sane defaults, and Postfix has forced me to still read them all just in case.

Worse, I can't count on the Postfix people not to introduce more insanity in upgrades (since it's clear that their ideas of sanity are significantly different from mine), so I'll get to periodically repeat this exercise more or less in perpetuity as long as I run Postfix.

So, in short, Postfix is not my favorite mailer because it has made it clear that I can no longer trust it.

(To be clear, I am still confidant that Postfix won't accidentally eat mail. I'm sure that all of the mail eating it may do to me in the future will be deliberate, set up by people who felt that it made perfect sense to eat email in that circumstance.)