Wandering Thoughts: Recent Entries

How I use FvwmIconMan

I've mentioned FvwmIconMan in the tour of my desktop and sort of mentioned part of how I use it, but I've never really explained the details.

As I've set it up, FvwmIconMan is essentially a compact taskbar for my various sorts of terminal windows. In a dense display, it shows the window name for each one (well, the first part of it at least), an indicator if the terminal has been iconified, and an indicator if that terminal has the keyboard focus. This is part of how I work around not having conventional titlebars on terminal windows; the window name information from the titlebar is dumped in small text in the 'taskbar', and through long experience I can pick out the label for the current window pretty easily.

(Possibly I should make the current window more distinctive than it is right now. A lot of my FvwmIconMan configuration, much like a lot of my fvwm configuration in general, dates from days with much slower machines that had much more limited graphics.)

Left-clicking on FvwmIconMan's label for a window toggles whether or not it's iconified. Like other taskbar implementations, an iconified (or 'minimized') window is only present as a label in FvwmIconMan; to deiconify it, I have to go click on the label. This means that I care a lot about finding the window labels for specific windows, and I do two things to help with this. First, the window labels are always sorted into alphabetical order; if and when a window is renamed, the order shuffles (this is very important for my use of xterm's ziconbeep feature). Second, I give my windows very consistent names based on either the host they're on or what I'm using them for (and sometimes both). This scheme usually works okay but breaks down a bit if I have a lot of iconified windows on the same host; usually I don't and this isn't an issue. Lots of non-iconified windows on a single host are generally not a problem because they're directly visible and I usually keep them straight by how they're arranged on the desktop.

(This alphabetical sorting does mean that the label for a particular window isn't in a consistent physical spot; it can jump around wildly depending on what other windows get named or renamed. This doesn't bother me, partly because a lot of my terminal windows come and go rapidly anyways. Non-alphabetical taskbars actually drive me up the wall because I never can find anything once I have more than a few things running, or at least I can only find them by scanning through the entire taskbar.)

Some taskbar implementations only show windows from the current virtual desktop or virtual screen or the like. While I use virtual screens I have FvwmIconMan configured to include all terminal windows, regardless of where they are. Among other things this lets me easily yank terminal windows between virtual screens; I move to another screen, then iconify the window and immediately deiconify it again (windows always deiconify on the current virtual screen) with two clicks on the window's label. I can also use FvwmIconMan to switch to the virtual screen that holds a particular deiconified terminal.

(Iconified terminals aren't on any particular virtual screen; they've been effectively swallowed by FvwmIconMan.)

Sidebar: terminal windows versus Firefox windows

A long time ago I would have confidently told you that I did this for terminal windows, and only for terminal windows, because they were by far my most numerous sort of window and I also often had a lot of them iconified. If I had the iconified windows represented as real icons on the root window, I would run out of space; therefor I condensed them all into a much more compact area. Then my Firefox window habit grew out of control and at this point I often have as many iconified Firefox windows as I have terminal windows.

So why do I have a taskbar for terminals and real icons for Firefox? The simple answer is that useful Firefox window names are too long, whereas I can make xterm window names short enough that I can pack them in very compactly. Because Firefox window names are long, a taskbar that showed enough of the titles to remind me what they were would be too big to be feasible. Instead it actually takes less space to have real icons and count on my spatial memory to remember what the Firefox icon over there is for.

(Well, the spatial memory plus the bit of the start of the window title that fvwm shows me below the actual Firefox icon.)

Why metaclasses work in Python

I've covered what you can do with metaclasses (1, 2, 3, 4) and even, sort of, the low level details of how they work (1, 2, 3). But I've never covered the high level view of why metaclasses work, ie what overall Python features make them go (partly because I am so immersed in Python arcana that much of that stuff feels obvious to me, although I doubt it actually is).

To start with, in Python everything is an object and all objects are an instance of something (yes, there are spots where this gets recursive). This includes even things that you wouldn't normally think of as objects, such as functions. Crucially, this includes classes: classes are objects. Any time you have an object in Python, a lot of its behavior is usually provided by whatever it is an instance of (to avoid confusion, I'll call this the type of the object). Classes are no exception to this; a lot of how classes behave is handled by their type, even things like how a new object gets created when you call the class.

(For simplicity, I'm going to ignore old-style Python 1.x classes from here onwards and assume that all classes are new-style Python 2 classes that ultimately subclass object.)

To avoid a point of confusion: classes have ancestor ('base') classes that they inherit from (or just object(), the root class). However, classes are not instances of their base class; we can see why this has to be when we note that a class can inherit from multiple base classes. You can't be an instance of several different things at once. So classes exist in a two-dimensional relationship; they inherit from one or more base classes, and at the same time they are instances of something that provides much of their 'class' behavior. The type of classes (the thing that provides the 'class' behavior) is called type().

(This two dimensional structure can get a bit weird.)

In some languages, the creation of classes is black magic that happens deep in the interpreter and isn't something you can do inside the language (even if the classes are visible as objects). Python has instead chosen to expose the ability to create classes by hand; you you can do this by calling type() with the right arguments (and then binding the class object to a name), just as you create instances of normal classes by calling the class itself. As part of creating classes yourself by hand, you can obviously manipulate class creation; you can create a new class with whatever methods, base classes, and so on you want.

(What's odd about type() is that despite it being a class, you can call it with a single object to get the type of the object.)

Python is also an unusual language in another way; in Python, things like defining functions and classes are themselves executable statements. Python doesn't parse your program, create all the functions and classes, and then start running your code; instead it starts running your code and things like def and class execute on the fly (as does import and so on). So it's natural to have your code running as classes are being created.

The combination of these two things means that Python can easily provide a way to hook your own code into the process of creating the class objects for classes that are written in straight Python, with 'class X(object): ....'. Python is already running code in general when this happens, and the mechanisms of creating classes by hand means it's relatively easy for Python to hand you the bits of the class-to-be so you can modify it and then have everything continue onwards to create a new class. This is why metaclasses can change classes as they are being created.

The other half of why metaclasses work is that Python allows classes to be instances of something other than type(). Since classes get a lot of their 'class' behavior through normal instance method inheritance from type(), a class being an instance of something other than type() lets the other thing intercept or change the normal as-a-class behavior for that class (for example, what happens when you call the class). This is why metaclasses can do things with a class after the class has been created.

The drawback of modern X font handling gets mysterious

Back in The drawback of modern X font handling I covered how modern X font rendering happens in the client and so can vary from client to client, going from nice on one client to bad on another. I illustrated this with xterm on Fedora and Ubuntu displaying the same font, Fedora well and Ubuntu badly. I now have a good reason to change to using xterm with modern fonts, so I spent part of today poking at this issue; the results have turned this into a genuine peculiar mystery.

What I have so far:

• the problem does not happen with all programs on Ubuntu. So far xterm and GNU Emacs have the bad font rendering, but Firefox, gnome-terminal, and TK-based programs such as exmh and tkmsg do not; they render DejaVu Sans Mono just like Fedora does.

• the problem only happens with some monospace fonts, not all of them. The Ubuntu machine I was testing on has 11 candidate fonts listed by 'fc-list :scalable=true:spacing=mono: family'; seven of them show the problem but four do not.

  (The good four are TlwgMono, Tlwg Typo, Courier New, and FreeMono. Unfortunately my preferred xterm font is DejaVu Sans Mono.)

• the problem is not the Ubuntu version of xterm, the Ubuntu app-defaults file for xterm, or even the Ubuntu Freetype library; I have built the Fedora xterm and my version of Freetype on Ubuntu and used the Fedora app-defaults, and the bad rendering is still there.

• I've directly set several fontconfig font rendering options that might be doing this without changing anything; at this point I haven't seen any difference with autohint, weight, embolden, or aspect (the last was a wild shot). Similarly, Xft X resources (cf) do nothing that I can see.

  (Forcing autohint=true actually makes the Fedora font rendering slightly but visibly darker while leaving the Ubuntu rendering unchanged for both the good and bad programs.)

• the problem doesn't happen with xterm on some FreeBSD machines I have handy; they render DejaVu Sans Mono the good way.

Clearly something mysterious is happening in the depths of the Ubuntu version of Xft or something it calls, but only if it's invoked in the right (or wrong) way. Unfortunately I don't think there's any good way for non-experts to see what font rendering choices are being made (the fontconfig library can be coaxed into some debugging output, but it's pretty much 'exports only' from what I can see), so I have no idea if I'll be able to figure out a solution that lets me use the font I want.

(Changing to gnome-terminal is not a solution for me.)

The death of system administration: I'm all for it

Recently there was a little Twitter commotion about Julian Dunn's Chef, devops, and the death of system administration (he later clarified his views). Although it may surprise people, my snap reaction to the idea of the death of system administration was 'good'.

(I have a number of other reactions to portions of this debate, but 'good' was my first one.)

Most of what many people think of today as 'system administration' is scutwork, at best boring and uncreative. Racking servers, configuring switches through interminable web or CLI interfaces, running network cables, installing OSes in any way that takes more than about one line of typing, writing an Apache or a mailer or Samba config file yet again, restoring files for people, and so on. That's what I'm talking about. At best these are interesting the first few times you do them; after that, very much not.

(System administration wasn't always this sort of work, but times have changed.)

Unless you really do like spending your time doing that or you feel that that sort of work is all that you have to contribute, you are better off without this near monkeywork. Regardless of what your job is called after 'system administration' goes away and the dust settles, you will have shifted to doing actual engaging and creative work and you'll be contributing much more to your organization's success. As I've written before in a different context, having spare time from ordinary day to day 'system administration' is what you need in order to create the big wins. The ultimate version of this spare time is not to have to do the ordinary day to day gruntwork at all.

As you may have gathered, I am not particularly fond of the scutwork currently involved in a great deal of 'system administration' (although I think there's uses for doing it every so often). As far as I'm concerned, the sooner this sort of system administration dies the better.

(At the same time, let's not fool ourselves. This death of system administration will put a significant number of people out of work, ie those people who are currently well paid to do nothing but this scutwork. Many of them do not currently have the skills to move up in the food chain; they will either move down to be less well paid operations monkeys or have to change fields entirely. This is going to be a wrenching process that will be very unpleasant for the people involved, and we should both have sympathy for them and understand the full implications of this shift we're casually discussing, advocating, and cheering for.)

(As a corollary, if you have junior people in your organization and you believe in this shift you should be working with them to make sure that they're developing the skills they'll need for the future, not just spending all of their time doing scutwork for you. And you should be honest with them about how you see their future.)

Why I use exec in my shell scripts

As with the little example yesterday, a fair number of my shell scripts end with running a program and when they do, I almost invariably go the little extra distance and do it with exec. In the old days, the reason to do this was that it used slightly less resources, since it got rid of the shell process and left only the process for the real program you wound up running. But, while I was around then, the reason I use it today isn't that; it's that it lets you freely edit the script while that final program is running.

At this point some of you may be going 'wait, what?' That's because most Bourne shell implementations are a little bit peculiar.

In most interpreted languages on Unix (like Python, Ruby, and Perl), the interpreter completely loads and parses the script file before it starts running it. This means that once your script has actually started running, once that initial load and parse has finished, you can freely change the script's file without the interpreter caring; it will only look at the actual file and its contents again if and when you re-run your script.

Bourne shell implementations have historically not worked this way (and it's possible that it's actually impossible to preparse Bourne shell scripts for some reason). Instead they not only parse the script on the fly as it executes, but also they read the file on the fly as the script runs. This means that if you edit a shell script while it's running you can literally shuffle the code around underneath the script. When the shell resumes reading and parsing the script after the current command finishes, it can be reading from partway through a line, from something that it had already read, or (if you deleted text) wind up skipping over something that it should have run. This often causes the shell script to fail with weird errors or, worse, to malfunction spectacularly. This can happen even if the shell is on the last line of the script.

But if you end a shell script with exec, you avoid this. The actual shell interpreter effectively exits (by turning itself into the actual program) and so there's nothing there to try to read anything more and get confused by your edits.

(Of course nothing helps if you can't use exec; then you just have to remember to never edit the script while it's running, at least with an editor that overwrites the file in place.)

Sidebar: a detailed example of what happens

Let's start with a little script:

#!/bin/sh
echo "a"
firefox

Run this script. While Firefox is running, edit it so that the echo string is four or five characters longer (using vi or some other editor that overwrites files in place). When you exit Firefox, the script will complain something like 'script: line 4: efox: command not found'.

When the shell was running Firefox, its read position in the file was just after the newline at the end of firefox. When you edited the script and added more letters, that same byte position was now pointing to the e in the 'firefox'. When Firefox exited and the shell resumed reading from that byte position, it read 'efox<newline>', saw a perfectly valid command execution, and tried to run 'efox' (and failed).

(It reports that this happened on line 4 because it knew it had already read three lines, so clearly this is line 4. As a corollary, you can't trust the line numbers that are printed when something like this happens.)

Every so often, I solve a problem with a hammer

For reasons beyond the scope of this entry, I maintain a special Firefox profile and instance for uploading pictures to my Flickr account. Back in the old days, Firefox had a very convenient behavior for this: when it asked you to choose files to upload in an upload form, the default directory was the current directory that you'd started Firefox in. This meant that I could cd to the day's photo directory, start my Flickr Firefox instance, and have the GTK file chooser dialog start in exactly the right directory. Then at some point Firefox changed this so that the default file chooser directory was something like your configured download directory.

I poked at this off and on but couldn't find a way to make Firefox get its old behavior back. So recently I decided to fix the problem with brute force. The script that I use to start my Flickr Firefox instance now looks somewhat like this:

#!/bin/sh
ln -nsf $(pwd) $HOME/CURDIR
exec firefox -P flickr "$@"

This is inelegant and not a real solution, but it makes things a lot more convenient; it's now much faster to navigate to exactly where I want to be. Sometimes that's the right way to deal with a problem, when either the real solution is too much work or the problem is too small to justify anything more than a quick hack.

(I suppose that this could be slightly improved by putting the symlink directly in the download subdirectory. I'm not sure why I didn't do that.)

My view of the purpose of object orientation

A while back I read Rise and Fall of Classic OOP. This caused me to realize that I am kind of a heathen as far as object oriented programming is concerned, probably because I came to explicit OO late and never actually learned how to do it the 'right way'. You see, to me object orientation is a technique for code organization and nothing more.

This gives me a very pragmatic view of when to write OO code and when not to; I use objects and classes where they make my code simpler, and I don't use them when they don't. I don't consider them something that has to be followed at all costs or as the only way to model the real world (or any arbitrary artificial world). If the real world entities that you're working with aren't amenable to being wedged into an OO hierarchy, then don't. Given the wide variety of both code structure and ways of organizing code so that it makes sense, it would be fairly absurd to say that OO is always the right answer; it is just one technique among many. Sometimes it's the right answer, sometimes not.

(Of course, some languages as so in love with OO that they don't give you a choice about it; you can't really have freestanding functions and data containers.)

I won't say that all of those OO examples that modeled the real world always struck me as a bit hokey and artificial, because honestly I never really thought that much about it (and any small example is hokey and artificial if you really look at it). But if people are switching towards my view of the purpose of OO, I'm all for it.

(I would be shocked if this was new and novel. I sure hope that lots of people have had this thought before me, because it just feels so obvious.)

The C juggernaut illustrated

Perhaps it is tempting, looking back at history from the vantage point of today, to say that C succeeded so much because it was at the right place at the right time. As you could tell the story, all sorts of people in the 1980s wanted a low level programming language, C was around, and so they seized on it. Any similar language would have done; it's just that C was lucky enough to be the one that came out on top, partly because of network effects.

(This story is especially tempting to people who don't like C and Unix.)

This significantly understates the real appeal of C at the time, even and especially to people who had alternative languages. A great illustration of this is C on the early Macintosh. You see, unlike environments like MS-DOS (which had no language associated with it, just assembler), the early Macintosh systems already had a programming language; they were designed to be programmed in Pascal (and the Mac ROMs were originally written in Pascal before being converted to assembler).

This was more than just an issue of Apple's suggested language being Pascal instead of C. The entire Mac API was designed around Pascal calling conventions and various Pascal data structures; it really was a Pascal API. Programming a Mac in C involved basically swimming upstream against this API, full of dealing with things like non-C strings (if I remember right, Mac ROM strings were one byte length plus data). I believe that Mac C compilers had to introduce a special way of declaring that a C function should have the Pascal calling convention so that it could be used as a callback function.

Despite all of this, C crushed Pascal to become by far the dominant programming language on the Macintosh. I don't think it even took all that long. Programmers didn't care that dealing with the API issues were a pain; working in C was worth it to them. It didn't matter that Pascal was the natural language to write Mac programs in or that it was a perfectly good language in its own right. C was enough better to displace Pascal in a hostile environment.

C did not win just because it was at the right place at the right time. C won in significant part because it was (and is) a genuinely good language for the job it does. As a result it was the language that a lot of pragmatic people picked if you gave them anything like a choice.

Another Russ Cox regexp article: How Google Code Search Worked

Russ Cox has just added another article in his series on regular expressions; this one is titled Regular Expression Matching with a Trigram Index, or How Google Code Search Worked. It's as worthwhile as all of the previous three.

How not to do repeated fields in web forms

There's a certain sort of web form which really wants to make sure that you've entered something correctly, so they ask you to enter it twice in two different fields. You've probably seen this in some web form sooner or later; this is the 'please enter your password again in this field too' or 'please re-enter your email address' field. I tend to think that this is bad on its own, but I've now seen an even worse implementation of this basic idea, which I'll call an anti-confirmation field, one that's practically designed to create errors.

What the people behind this did was quite simple: they made it so that their second fields would not accept pasted input (probably using JavaScript, which I had on because I didn't feel like finding out which bits of the registration process required it). I had to retype both my email address and my password by hand, which was especially annoying because I was pasting both of them from elsewhere. I call this an anti-confirmation field because of course retyping things by hand is more error-prone than pasting things in; in fact, I twice made a mistake retyping the password.

(My web password for this site was a strong random password, as usual. Random jumbles are hard to transcribe accurately by hand, especially when they jump back and forth between character case.)

I suspect that the website designers justified this by saying that they were worried about people entering a bad email address by hand in the first field and then 'confirming' it by just cutting & pasting it into the second field. However, even at its best this logic doesn't work for password fields since browsers don't let you copy the plaintext content of a password field once you've entered it. I also suspect that the designers do not have any actual data on how many genuine errors this prevents (versus how many artificial errors are created).

Sidebar: how to measure the numbers

Assuming that you've committed yourself to (anti-)confirmation fields in the first place, you just need to track field values across time when a submission fails because of mismatched fields. In a transcription error the first of the two fields will turn out to be correct (ie, the same as the final submitted value) and the second field will change. In a genuine error the first field will be different between the failed submission and a subsequent valid one.

Doing this with email addresses raises basically no security issues. If you do this with the password field you'll want to one-way hash them somehow in your tracking data.

Let's make it official: Solaris 11 is closed source

You may remember back in August 2010 when there was a leaked Oracle memo that said, among other things:

We will distribute updates to approved CDDL or other open source-licensed code following full releases of our enterprise Solaris operating system. [...]

At the time I noted that 'full releases' might be construed to be 'Solaris 11' instead of the next 'Solaris 10 update X' release and was unhappy about it. That was then. Now it's been a couple of months since Solaris 11 was officially released to the world and, well:

; cd onnv-gate
; hg incoming
comparing with [...]
searching for changes
no changes found
; hg log | fgrep date | sed 1q
date: Wed Aug 18 15:52:48 2010 -0600

I think it's safe to conclude that there will be no further updates to public (Open)Solaris source code from Oracle, ever. Solaris is now a closed source, 'source-not-available' operating system once again (and probably stronger than it ever was; it used to be sort of possible for universities to get Solaris source code, but I doubt that's on the table from Oracle).

(I'm sure that almost everyone concluded this some time ago. Sometimes I remain hopeful even in the face of all but certain disappointment.)

This matters a lot for us; our ZFS spares system and parts of our ZFS status monitoring system are built around information obtained from undocumented internal library interfaces because there is no other alternative. It seems extremely unlikely that we will ever upgrade to any future version of Oracle Solaris. Lack of (Open)Solaris kernel code also significantly reduces the usefulness of DTrace, one of the theoretical signature Solaris features.

(It is vaguely possible that some version of Solaris will sometime expose public interfaces for the information we need, but frankly I really doubt it. All evidence to date suggests that it is strongly against the engineering culture of ZFS; they had five years to do it and steadfastly didn't. I will skip any number of angry remarks.)