Wandering Thoughts archives


A rule of thumb: Automate where you can make mistakes

One of my sysadmin rules of thumb for deciding what to automate in scripts and programs is this: automate where you can make mistakes. In particular, automate where you are specifying redundant information.

This will make more sense with an example, so let's talk about configuring iSCSI targets. We use static target configuration, which means that you tell the system a target name and the IP address that it can be found on; each iSCSI server has two IPs, so we configure each of its targets twice, once for each IP address.

(An iSCSI server machine can have several targets, each of which has several LUNs. In our environment, each iSCSI target represents a single physical disk, with the target names divided into a per-host and a per-disk portion.)

There's an obvious redundancy here; we know for sure that server A's targets are never going to be found on any IPs besides those that belong to server A. But when we specify this redundant information by hand, we allow errors to creep in; we could accidentally configure one of A's targets with an IP address for server B. (And indeed we did this once, and it turned out to be surprisingly difficult to sort out.)

So we automated the process of adding iSCSI targets to make sure that we couldn't make this mistake, or other related ones (failing to configure each target for both IP addresses or failing to configure all of a server's targets). Our program for this now just takes the server's name; from this it can determine all of the server's targets and both IP addresses for the server, and add all 24 separate static target entries for us.

Such automation is clearly not general; instead it relies on very specific knowledge of how we always set up our systems. But since we do have strong conventions about how we set up iSCSI targets, we might as well exploit them (in a script) to avoid errors and to make our lives easier.

(And if we ever have to break our conventions, well, we can always use the underlying system commands directly. Escape hatches are important too.)

sysadmin/AutomateToAvoidErrors written at 22:48:22; Add Comment

Never kill the screen locker

This is a grump.

Dear X applications (or any application on any window system): you should never, ever kill or otherwise force-terminate the screen locker. In particular, you cannot assume that just because you have been run that the user is sitting there in front of the screen and wants it unlocked. There are any number of ways that you can start up when the user is not present, and unlocking the screen in this situation can easily make things go horribly wrong.

For one example, perhaps some resource such as an NFS filesystem wasn't available and your program hung as it was starting, so the user gave up, locked their session, and walked away from the computer to do something productive. When the NFS server recovers, your NFS IO suddenly starts working, and you actually start up, the user is going to be very peeved to come back to their machine being unlocked.

(I'm pretty sure that I've seen KDE applications do this. The situation that caused it aren't clear to me, since I wasn't in front of the machine at the time, but I believe that the application was either trying to show a splash screen or trying to show a 'I have run into some problem, do you want to send a bug report' note.)

I'm sure it's tempting to decide that your program has something so important to communicate to the user that they need to see it right now and you need to make it visible. However, you're wrong; it can wait until we know that the user is present because they've just typed in their unlock password.

(The sole exception to this is if the system itself is about to crash or reboot, and thus your message literally cannot wait because it's about to disappear in a short time no matter what. But you'd better be very sure that you're not about to leave the user's session unlocked for any more than a very short time.)

Sidebar: starting an X application remotely

Suppose that you are ssh'd in to your workstation and you want to start an X application, displaying on your workstation's screen. The easy incantation to do this (in Bourne shell) is:

DISPLAY=:0 x-program &

You may need to do something to get the program to live happily in the background, and you may have to specify where to put the program's window (depending on your window manager and the state of your system at the time; many window managers now auto-place new windows without user intervention).

(In theory many X programs accept a command line argument to specify the X display to talk to, but with so many toolkits and systems these days, who knows what command line argument any particular program accepts and what restrictions it has. Setting $DISPLAY is much simpler and it works for everything.)

programming/NoKillingScreenlocker written at 02:03:49; Add Comment

Page tools: See As Normal.
Login: Password:
Atom Syndication: Recent Pages, Recent Comments.

This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.