Wandering Thoughts: Recent Entries

The XFT font naming issue

One of the subtle problems with XFT fonts is that they don't have canonical, broadly supported textual names. To explain this, I need to go back to old server side XLFD fonts to show what I mean and the difference between XLFD fonts and XFT fonts here.

Fundamentally XLFD fonts are specified by their name, right from the relatively low level library APIs on up. You get a handle to the font by asking for it by name, and the name you use is its standard canonical format XLFD name. This means that everything that deals with XLFD fonts fundamentally uses a standard textual identifier for them, and in turn this means that everything accepts the same name to specify a particular XLFD font.

(Now, there are aspects of this that are less than ideal. For instance, a program that starts with one font and wants to derive a bunch of closely related ones (eg the same font in different sizes or the same font in bold or italic) is going to be parsing the XLFD name for the font into its component parts, changing some of them, and then reconstituting this into new font names.)

XFT fonts do not work like this. While XFT fonts sort of have a standard format name due to FontConfig (cf the font names section), these names are not actually a fundamental part of how XFT fonts are used and specified; instead, XFT fonts are usually specified as a collection of attributes and properties. Your code does not ask for a font with the name 'Monospace-14', it asks for a font with family Monospace and point size 14 (and so on). How you go from a text name for a font to this collection of attributes is to a certain extent up to your program in a way that it was not for XLFD font names.

(This is of course convenient for programs that want to deal with a cluster of fonts; now they can just directly vary the attributes they care about, like the point size.)

What this has created is a situation where different programs can each have a different way of naming the same XFT font. Since it's to a large extent up to each program to turn a font name string into the collection of XFT font attributes that it will actually use with the API, each program can and often does do it differently. There's no guarantee that two different programs will accept the same XFT font name and there's no guarantee that a particular program will accept all of the theoretical features of XFT font name specifications that you'll find outlined in the Font Names section of the FontConfig user documentation. Programs are in fact free to be quite limited in what they accept (the practical minimum is family and point size, but even then the format can and does vary). You have to check the documentation for each program to see how it handles XFT font names, what format they're in, and what you can and can't put in a name. And the answers vary, a lot.

What this means pragmatically is that XFT fonts are more annoying to deal with than XLFD fonts if you're primarily specifying them by (text) name. In the XLFD era, you had a single name for a font and everything accepted it. With XFT fonts, you have multiple names for the same font and you have to work out how to format the name of the font so that today's program will accept it (if it will at all; if you need to specify advanced options, it may not). Also, some programs may require you to specify the font in multiple bits (they may have a separate argument or setting for the family and the font size, for example, instead of letting you put it in one name).

A semi-brief history and overview of X fonts and font rendering technology

Because I've just been mucking around in this swamp, I feel like writing all of this down.

In the very beginning, X had some simple bitmap fonts in the server with equally simple names that were basically labels; ie, the X server attached no particular meaning to the font names, which were created by people mostly to be short and sometimes to be vaguely meaningful. Some of these font names linger on in X today; the most well known one is fixed, the name for the default monospace font (and font size), but there are others like 6x10 and 6x13bold. The X clients just asked the server to display some text in the named font, and the server handled all of the rendering and drawing.

(Today these font names are implemented as aliases for other fonts.)

As the X server evolved, it grew more bitmap fonts. Fonts from different vendors, fonts in more sizes, proportional fonts as well as monospaced ones, fonts for different resolutions (75dpi versus 100dpi), fonts for different character set encodings, and so on. It was clear that ad-hoc font names weren't going to scale because no one was going to be able to keep fonts straight or find one. So the X people invented a naming convention for their fonts, the X logical font description (XLFD). In theory an XLFD name describes most of the important attributes of a font, things like the point and pixel size, the slant, the style, whether it's proportional or monospaced, and so on. Along with XLFD names and their defined structure, the X people introduced the idea of wildcard matches so that X programs could say 'I don't really care which vendor it comes from, I just want whatever 15 pixel monospaced font you have'.

(For backwards compatibility, the original simple X font names were defined to be acceptable XLFD font names (although you can't use wildcards with them).)

Initially XLFD fonts were still all bitmapped fonts, just better named and more numerous than before. However the X server soon got additional font rendering support that let it handle several scalable font formats of the time. Scalability was implemented in XLFD font names in a straightforward way; if the font name itself had zeros in various resolution related fields, you knew that the X server could and would render it at whatever pixel size you asked for.

At some point, people observed that the precious X server was spending a bunch of time and memory loading, parsing, rendering, and so on all of these fonts, even for fonts that weren't very actively used. So the X people decided to offload a lot of this work to a separate daemon, the X font server (xfs), although in the usual manner of X they left all of the direct server side font rendering code intact too. An X font server could (in theory) be isolated from the X server and if necessary restarted independently; all of the hairy and CPU consuming code for scalable font rendering could be parked in a process in it, and so on. The X font server could handle all font formats that the X server could, and thus could render all XLFD fonts for the server. Although the X server delegated actual font rendering to another process, X font handling was still done in the server; clients still just told the server 'draw text X in font Y'.

(Generally xfs could be pointed to exactly the same font directories as the X server had previously been using directly.)

The next evolutionary step in X font handling was to move it to the client side, which marked (and marks) a stark division in X font handling. This is XFT and 'XFT fonts'. XFT is to a significant degree glue; it uses FontConfig to translate from font names (and attributes) to actual concrete font data files, then FreeType to turn text into picture data and draws the picture data using various X bits.

Technically and theoretically XFT and its pieces still support old X bitmapped fonts. Practically they do not; XFT and XFT-using programs really expect fully scalable fonts, generally ones with a wide glyph selection, and have basically no patience or tolerance for bitmapped fonts that are available in only a few point sizes with only a few glyphs. With heroic work in FontConfig configuration files you can sort of get something limping along, but in practice moving to XFT fonts means no more bitmap fonts.

(Yes, I have tried this experiment. It's especially unsatisfactory for 'frankenfonts', ones where the real font is only available in a few pixel sizes and you were already filling in other pixel sizes with substitutes. The XLFD configuration system is much better for this.)

Generally, the system FontConfig configuration will look for fonts in all of the X server font directories with scalable fonts, or at least all of the directories that are considered to include 'good' fonts. This makes scalable XLFD fonts available to modern XFT-using clients, although under somewhat different names.

(TrueType fonts will generally render the same in XLFD and XFT form because the X server and the X font server long ago were set up to render them with FreeType. Any remaining differences in appearance are due to rendering decisions made differently between FontConfig and the X server environment. I'm not sure how older scalable font formats come out, and generally you don't want to use those fonts anyways.)

So today X has two separate font technologies: XLFD fonts and XFT fonts. XLFD fonts are configured through xset's font path directives (the modern coolness is catalogues) and perhaps your xfs configuration file (if xfs support is still working in your X server, which it probably isn't). XLFD fonts and font directories exist only on the X server machine (or perhaps the xfs machine) and properly set up font directories have fonts.dir, fonts.scale, and/or fonts.alias files that describe their contents. These are the only bitmap fonts still supported and XLFD bitmap fonts usually work well at small pixel sizes.

(This is especially true of monospaced bitmap fonts, many of which were extensively tuned for high readability and high density at relatively small pixel sizes.)

XFT fonts are configured through FontConfig. Magic happens, and it happens differently on different machines. On the good side, you can generally put new font files into $HOME/.fonts and they will automatically be recognized for you. FontConfig has various sorts of support for font aliases, but they hide out in magic places like the XML configuration files in /etc/fonts (if you want to see some of it, look for how many things in /etc/fonts/conf.d want to lay claim to be 'monospace', the default monospaced font). Most XFT fonts are TrueType fonts (at least the modern ones that you want to use) and do not work well at small pixel sizes.

XLFD font support is pervasive in old X programs but scattered and increasingly absent in modern ones. XFT support is the inverse; uncommon in old X programs and old Unixes (such as Solaris), but increasingly common in modern X programs (especially ones that are part of a pervasive environment like Gnome or KDE, where it is ubiquitous), on Linux, and at least partially on other Unix OSes such as FreeBSD. Some programs and frameworks make an effort to support both XLFD fonts and XFT fonts, but many are XFT-only.

(Today's expedition into this swamp was started by Tk 8.5, which can be compiled to support XLFD fonts or XFT fonts but not both at once. You can guess which option modern Linux distributions have picked.)

Bash's superintelligent errors about exec failures

Let's take a closer look at bash's error message from yesterday, because if you pay close attention something really interesting is going on. Here's the error message again:

bash: /tmp/exmpl: /bin/shx: bad interpreter: No such file or directory

On first blush, if you don't think about it too much, this looks perfectly sensible: bash is reporting that when it tried to exec() the /tmp/exmpl, the kernel told it that there was a problem with the script's interpreter.

But, wait. When the exec() fails, all that kernel can tell bash is the errno number. In this case the kernel returns ENOENT, which creates the 'No such file or directory' portion of the error message. So how does bash know that the reason that trying to run /tmp/exmpl failed is because /bin/shx doesn't exist?

Here, have another example error message from bash:

bash: /tmp/a.out: /lib64/ld-ZZZZZ-x86-64.so.2: bad ELF interpreter: No such file or directory

(I did some binary editing to create that failure.)

That's right. When an exec() fails, bash opens the executable and parses it to try to identify what went wrong. It recognizes shell scripts, which is easy, but it also parses ELF binaries to find things like the name of the ELF interpreter, so it can check that. Let me say that again: bash knows how to parse ELF binaries so that it can give you good error messages. I must applaud bash's attempt to be almost as user friendly as possible, but at the same time I think it went more than a little bit overboard.

(If you trace the system calls it's using, you can clearly see it selectively reading several bits of the ELF binary.)

If you try these same things in other, simpler shells, they will simply report something like '/tmp/exmpl: No such file or directory', ie they are simply doing a straightforward translation of what the kernel told them (even if it is a rather puzzling message).

(zsh will report the interpreter problem with a shell script but not with a binary, which strikes me as a reasonable amount for a complex shell to do. Checking for this error with shell scripts is not that difficult and it does happen periodically.)

Why you should never use file (or libmagic) to identify files

Every so often, someone needs their program to figure out what sort of thing a file is; is it text, or HTML, or a JPEG image, or Postscript, or whatever? When this happens it must be very tempting to use the file program to classify things, especially since some versions of file will give you a MIME type for the file (instead of just a text label).

Here, presented in the traditional illustrated form, is why you do not want to do this:

; file example
example: Netpbm PGM image text
; cat example
P238: An introduction

Lorem ipsum dolor sit amet.

File is exceedingly generous with classifications. It does not verify that your target file contains anything like a valid instance of the file type; instead, it checks for signatures. Over time, lots of people have added lots of signatures for lots of file formats. A certain number of these signatures are very minimal and so will match lots of things. This creates misclassifications where unknown file formats and plain data can match a minimal signature if things are just right (or just wrong, from some perspectives). People and programs who use file to identify and classify files are operating under a mistaken impression of what it really says. File does not say 'this is definitely a <whatever>'; instead it merely says 'this kind of looks like a <whatever> to me'. The difference is important.

Some of you might think that this is theoretical and will never come up in real life. I regret to inform you that our CUPS print system just did this to someone, causing their plain text files to get fed to an image converter (which choked, meaning no printouts for this person).

(CUPS is probably not literally running file, but these days file is just a wrapper around the libmagic shared library. Which exists so that people can use it for exactly this purpose, sadly.)

Note that this is not merely a Linux issue. The version of file on, eg, a not all that current FreeBSD machine will also misidentify this plaintext file as a Netpbm PGM image.

What it means to become another user on Unix

Ignoring things like SELinux for the moment, the basic security state of Unix system calls has always been that root is allowed to become any other UID at will, but no one else is allowed to change their UID or other security attributes (setuid programs then provide an escape hatch from this). But what does it mean to become another user on Unix, beyond just setting your UID? In fact there are a whole series of things that it can mean, some of which you do not necessarily want.

Let's make a list:

• switching to the user's UID. This is the basic prerequisite of what it means to become another user on Unix, and is done by setuid().
• switching your groups to the user's groups, which is done with a combination of initgroups() and setgid().

• setting some environment variables like $HOME and $SHELL to values appropriate for the new user.

  (Most but not all versions of su do this. Versions of su that do not change $HOME can be exciting, especially when combined with shells that read initialization files from $HOME.)

• clearing various environment variables; in the extreme case, you will clear all environment variables and give yourself a new set of safe values for things like $PATH and so on.

  (Aggressively scrubbing the environment is generally the default for sudo. In general there are any number of environment variables that are very dangerous to leave intact, such as all of the LD_* variables that influence the behavior of the dynamic loader.)

• running the user's shell, either interactively or with -c ... to execute commands.

  (su normally always runs the user's shell; sudo will often run commands directly without starting the user's shell.)

• changing to the user's home directory.
• running the user's shell as a login shell.

• allocating a new pseudo-tty as the user and attaching the user's shell and so on to it as a new session. This is generally the domain of things that are making real login sessions, like sshd, instead of programs that just become the user.

(Note that this list is not in the order that your code wants to actually do these operations. For example, you want to set groups before changing to the other UID because the moment you setuid() to a non-root user you lose the power to set your own groups list.)

Many things that become another user deliberately run the the target user's shell even when they just want to run a command. This is done partly so that the user's shell can do any special initialization that, eg, may augment its $PATH and partly so that users with restricted shells can't escape them in various clever ways. However, not everything does this and sometimes running the user's shell is inconvenient.

There are a surprising number of cases where you want basically the first two only; you want to run a command as a user but in a way such that their environment (their shell, their shell initialization, and so on) is ignored. It's possible to do this with standard commands like su and sudo by carefully reading their manpages and picking exactly the right options, but I find it easier to have a very basic runas program sitting around.

(runas only works for root, so it can completely ignore authentication and similar issues.)

My view of where the Unix community is

I've written before about how there once was a time when Sun could be seen as the heart of the Unix community (cf), and this view of Sun's place in the community is why some people don't like Oracle's Sun-related actions (cf the commentator here). As I wrote back then, my feelings about Sun are somewhat more complicated and tangled (cf) because I don't think that Sun has been the heart of the Unix community for a long time.

There's a lot of reasons that Sun stopped being the center of the Unix community (and maybe I'll write about them sometime), but the important thing is that it did not happen this decade or even this century. Sun ceased being the center in the mid 1990s, when it increasingly had only tired hardware and tired software to offer. Even back then, it was not hard to see where the community was going if you paid attention: it was going to x86 systems, because they were good enough and cheap.

(I don't have a clean and clear definition of the Unix community, but to me it is some mixture of the people using Unix, the people enthused about Unix, and the people who are evolving Unix and moving it forward. Mostly the latter two, really; you need to be interested in Unix as itself in order to really be part of the community instead of just a passive user of computing. This is why most of the OS X people don't count.)

Thus it's clear to me that today's Unix community is found in either FreeBSD or Linux. The part of the community that likes Unix as it is has mostly wound up in FreeBSD; the part of the community that has radical dreams for the future has mostly wound up in Linux (this is where it's appropriate to borrow a Henry Spencer quote and say 'those who don't understand Plan 9 are condemned to reinvent it, poorly'). This is not a new development; it's been this way since no later than the late 1990s.

(My somewhat inflammatory view is that while many of the people of the old Unix community who stayed around Unix seem to have wound up in the *BSDs, the spirit of the Unix community mostly settled in Linux. Unfortunately there are significant drawbacks to this split, some of which I have captured in my borrowed witticism.)

Sun's death and absorption by Oracle was not a real loss for the Unix community because Sun hasn't been important to the real Unix community for some time. The real Unix community is going on just as it has been for years, and is probably roughly as healthy as it ever has been.

Sidebar: why the Unix community is not in all of the *BSDs

OpenBSD is not where the Unix community is today because its scope is too small, although valuable. NetBSD and DragonFly are simply too small, period. Only FreeBSD is large and encompassing enough.

Part of the cleverness of Unix permissions (a little thought)

Ever since it's become popular to add various sorts of advanced permissions schemes to Unixes (ACLs, for example), it's struck me that part of the genius of Unix permissions is that they are short.

I don't particularly mean short in storage terms (although that didn't hurt Unix in the early days). I mean short more in the sense of 'simple'; Unix permissions are simple enough that they have a short, clear representation. You can describe the basic access permissions for a file with three relatively short strings (the actual permissions plus who the owner and group of the file is), and these strings are mostly self-contained.

(You don't necessarily know who is in the group without running a command.)

I think that this shortness is an important part of why Unix permissions work relatively well. Short permissions are easy to display (which means that they can be displayed routinely, for example in 'ls -l' output) and relatively easy to understand. Because they are simple, they have few surprising interactions. I'll even go so far as to say that short, simple permissions are relatively easy to manipulate.

(There are aspects of Unix permissions that are not easy to understand and predict, things like what even the permission bits mean on directories, what the various obscure permission bits mean on everything, and so on. But I tend to think that this just adds complexity around the edges, not in the center.)

Some things about changing from old X fonts to modern TrueType fonts

Now that I've transitioned to using modern fonts in xterm, I've decided to move into the 00s by changing to a UTF-8 based text encoding (I've considered this some time, but only once I had more or less Unicode-enabled fonts did it become actively tempting). Making this shift provoked a small pile of complaints from FVWM about my bitmap fonts not having all of the font charsets that a true Unicode environment wants, which caused me to start switching various other fonts to modern TrueType fonts.

One of my challenges in doing this was matching the size of my new fonts to the size of my old fonts, since I didn't want to significantly change the size of things in my desktop environment; shuffling fonts is bad enough without shuffling layouts as well. Fortunately there turns out to be a relatively easy trick for this that mostly works.

There are two ways to specify the sizes of old-style X fonts; you can use either actual pixel size or a point size (or both at once, if you're so inclined). If you only have the point size, you need to determine the pixel size. The easy way to do this is to fire up xfontsel like so:

xfontsel -pattern "-adobe-helvetica-bold-r-normal--*-120-100-100-*-*-*-*"

Now find the pxlsz dropdown, and it should normally have exactly one pixel size (in this case, 17 pixels).

Once you have the pixel size you can simply directly specify this in an XFT font specification such as 'Sans:Bold:pixelsize=17'. This will give you a font that is either exactly or almost exactly the same vertical size as your previous old X font. I don't know if there's a magic XFT way to work out the correct (point) size that corresponds to a specific pixel size; so far I haven't worried about it.

(The font rendition may not be the same, of course; in my case I found that I needed 'Condensed Bold' instead of plain 'Bold' in order to match reasonably well. You're going to have to experiment.)

The other thing I found out after some experimentation is that modern TrueType fonts are not as good at small pixel sizes as the old X bitmap fonts. After playing around with it I've stuck with my old X fonts for small sized things (an example is the places where I'm using the classic 8x13), complaints from FVWM and all.

(It's possible that someday we'll have modern TrueType fonts that are very well hinted for such small sizes, but they don't seem to be there today. It's not glaringly bad by comparison, but my FvwmIconMan definitely felt less readable and clear when I was trying out 'Mono:pixelsize=13'.)

Sidebar: Xt and bad font rendering in UTF-8

One of the things that happens in some Xt-based applications, most visibly xclock, is that if you use a UTF-8 based text encoding things suddenly insist on doubling the vertical space they use for rendering each line of text. I don't know why they do this, but so far my only solution is to turn off LC_CTYPE when I start those applications so that they revert to the default C locale.

Understanding FVWM States, with better FVWM code for selecting terminal windows

Every so often I make an epic mistake, where I drastically misunderstand something. A discussion with one of the people behind FVWM in response to yesterday's entry on selecting terminal windows from the keyboard revealed that I had made one of them about an FVWM feature called (window) 'States'; a correct understanding of the feature leads to a better version of the FVWM code from my last entry.

What FVWM States are is effectively user-controlled tags for windows, given numbers from 0 through 31 instead of (say) text labels. Crucially, States are completely independent from each other; a given window can be in multiple States at once, and individual States are manipulated independently. Windows default to having all States turned off but can have this changed by FVWM Style commands or on the fly. States can be used for at least two separate things. The first is to keep track of actual window states, such as whether or not a window has ever been iconified; the second is to aggregate windows together into some sort of abstract category such as 'all terminal windows'.

(My mistake was thinking that a window could only be in one State at a time, which meant that using States to identify terminal windows would preclude using States for anything else involving them, more or less. This is wrong.)

The second sort of use of States is the interesting one for what I was doing in yesterday's entry, so here's revised FVWM code with commentary. First, we mark all windows we consider 'terminal windows' with a specific State through a Style command. I've picked State 2 here, but the choice is arbitrary (provided that you aren't already using the state number).

Style "XTerm"            State 2
Style "9term"            State 2
Style "Gnome-terminal"   State 2

Any time your FVWM configuration wants to specify 'this applies to terminal windows' it can now simply use 'State 2' as the matching condition. Specifically we can do this in my ToWindow function:

AddToFunc ToWindow
+ I   Current (State 2, "$0") Break
+ I   Next (State 2, "$0") ToWindow2

AddToFunc ToWindow2
+ I   Iconify False
+ I   Focus
+ I   Raise
+ I   WarpToWindow 80 20

Using a State to identify terminal windows has the great advantage that if you start using another form of terminal window, such as Konsole, you don't have to go through your FVWM configuration to add additional '|Konsole' matching conditions to everything; you just add a single Style declaration to identify Konsole as a terminal.

Style "Konsole"          State 2

(If you actually want all terminal windows to have the same window manager decorations and so on, you can do even better than this. FVWM allows for abstract styles, so you can create a 'terminal' style that decorates the window appropriately and also includes the 'State 2' setting, then simply say 'Style "Konsole" UseStyle terminal'.)

(I'd like to thank Thomas Adam for taking the time to patiently get enlightenment into my head.)

How I select (terminal) windows from the keyboard in FVWM

As part of reorganizing my work desktop to deal with big screens, I decided that I wanted a way to select terminal windows by name from the keyboard, with some form of autocompletion for speed. The goal was to give me a faster way to get a particular terminal window than sweeping my mouse all the way to my taskbar for terminals and finding the specific window I wanted. FVWM doesn't have any native features for this, so I had to bolt something together for this out of pieces.

(To make life simple I decided to not do anything clever about several windows having the same name. If that happened and I picked the name, the system could pick whichever specific window it wanted.)

The most difficult bit is selecting among alternatives from the keyboard with some form of autocompletion. Fortunately I already had a general program that could do this: dmenu, which I was already using as the core of a general 'do things' launcher. Dmenu will read a list of things from standard input, put up a menu/text entry area, let you type to it, and then print out the selection to standard output. All I needed was a way of generating the list of terminal window names and then of making FVWM activate the chosen window.

With the help of the FVWM mailing list I managed to put together both parts of this. Here is what I have. First, the driver script (slightly simplified; the real version also matches gnome-terminal):

#!/bin/sh

# Generate a list of unique names of
# terminal windows.
genwins() {
  xwininfo -root -tree |
  egrep '': \([^)]* "(XTerm|9term)"\)  [0-9]' |
  awk '{print $2}' |
  sed -e 's/^"//' -e 's/":$//' |
  sort -u
}
# run dmenu to get the answer.
win=$(genwins | dmenu -b -p win -P -t) || exit 1
# pass the selected window to FVWM
echo "Function ToWindow \"$win\"" |
  FvwmCommand -c

(The -P and -t arguments to dmenu are from a personal patch that I did. -t is especially useful because it makes dmenu do shell-like partial autocompletion. The real script also specifies the dmenu colours and font to use. There may be a more elegant command than xwininfo to use for getting the window names, but xwininfo has the virtue of being ready to hand.)

In FVWM, I've defined the ToWindow function as follows (the real one also selects gnome-terminal windows):

 AddToFunc ToWindow
 + I   Current ("XTerm|9term", "$0") Break
 + I   Next ("XTerm|9term", "$0") ToWindow2

 AddToFunc ToWindow2
 + I   Iconify False
 + I   Focus
 + I   Raise
 + I   WarpToWindow 80 20

Update: there is a better way to write ToWindow using FVWM States; I've written it up in FvwmStatesUnderstood.

This does nothing if either there is no such terminal window or the current window is a terminal window with the right name. Otherwise, the named terminal window is deiconified, given the focus (which also switches me to its virtual screen if it's not on the current virtual screen), raised to be on top of everything else, and then the mouse pointer is moved into it. I don't normally like having the mouse pointer moved around on me, but it's necessary here in order to make sure that this window keeps the focus; popping up a window and then immediately having it lose focus was more irritating than having my mouse pointer teleported.

(The odd structure of this as two functions is necessary for internal FVWM reasons. Basically it's the easiest way to insure that nothing happens if there's no such window.)

(I also had to add FvwmCommandS (the FvwmCommands server module) to the set of FVWM modules that my configuration runs, since I hadn't been using it before this.)

PS: This code omits '|Gnome-terminal' in three places purely to shorten the length of the unbreakable lines in this blog entry.