Wandering Thoughts archives

The old-fashioned and modern ways to remap keys in X (some notes)

As I mentioned recently, I've been remapping my keyboard in X for a very long time. In all of this time, I've been using the traditional and now old fashioned way of doing this, namely xmodmap. Xmodmap is now a deprecated mechanism, although it still works; these days you usually want to be remapping keys with setxkbmap.

You might wonder why X has two different programs for this. As you could guess, it's the usual reason; X added a new system for handling keyboard remapping without removing the old one (because X never removes anything). Xmodmap uses the old system, which I believe has been in X since at least the start of X11, while setxkbmap uses the new system, the X KeyBoard extension.

(Well, 'new' is relative here. XKB apparently dates from 1996, almost two decades ago. I was a little bit surprised to find that out, since I hadn't heard of it myself until recently. This is probably the hazard of having become set in my X ways a very long time ago.)

I'm not the right person to write up a big evaluation of the two mechanisms against each other, because up until recently I hadn't looked into XKB and setxkbmap at all (and even now I only know a bit). So from my limited perspective, the big difference between the two is that xmodmap offers relatively simple and direct low-level control for weird operations while setxkbmap gives you simple ways to do most of what you often want, provided that you can work out the right arguments.

To show what I mean by this, here is more or less the xmodmap magic needed to swap Backspace and Delete keys and to make CapsLock act as a Control key:

keysym BackSpace = Delete
keysym Delete = BackSpace

remove Lock = Caps_Lock
keycode 0x42 = Control_L
add Control = Control_L

You put this in a file and then run 'xmodmap FILE' and magic happens.

By contrast, here is the setxkbmap command needed to make CapsLock act as a Control key and make the right Windows key act as Compose:

setxkbmap -option ctrl:nocaps -option compose:rwin

You will notice that I have not shown you how to swap Backspace and Delete with setxkbmap. That's because that particular change is not among the predefined changes that XKB supports and once you venture outside of those predefined changes, things become rather complicated.

(The predefined changes themselves are documented in the xkeyboard-config manpage. The files that define what these options and so on mean are all found under /usr/share/X11/xkb; see for example /usr/share/X11/xkb/symbols/ctrl. If you look at these files, you'll see that XKB configuration is, well, intricate.)

Because setxkbmap fits my needs right now (now that I'm not swapping Delete and Backspace) and because it's apparently the more approved-of way these days, I've switched from xmodmap to setxkbmap. I probably would not have changed over if I was doing anything that setxkbmap couldn't handle through its existing options for -options; while more sophisticated things are possible, they appear to be a lot of work.

The ArchLinux wiki has some nice references on Keyboard configuration in Xorg, xmodmap, and the X KeyBoard extension. While I accumulated a bunch of additional links while I was looking into setxkbmap, I'm not going to put any of them here because I don't know enough to assess their quality and whether they're up to date.

The great Delete versus Backspace split

In the history of Unix, one of the great quiet divisions has been the split of what your erase character should be, with the choice being DEL (Ctrl-?) or Ctrl-H. In turn, this goes back to the physical serial terminals that for years were how you logged on to most Unix systems. In the days of real terminals, you wanted your Unix erase character to be set as whatever your terminal generated when you hit the big, prominent 'backspace' key (which was, of course, not at all programmable). Setting DEL when your key generated Ctrl-H or vice versa was a recipe for frustration and extra work.

If life was great, everyone would have agreed that the terminal backspace key generated one thing and we'd be done. Of course that didn't happen. As I remember it, most of the world's serial terminal makers decided that the backspace key would generate Ctrl-H, but Digital (aka DEC) decided to be special and have their backspace keys generate DEL. The Digital VT-52, VT-100, and later VT-series serial terminals were very popular, so Digital's decision had an outsized effect on Unix users. Adding to the fun for Unix users was the GNU Emacs decision to bind Ctrl-H to 'get help' and officially bless DEL as the (sole) delete character.

(It didn't hurt the (Unix) popularity of the VT series that for years you bought Digital computers to run Unix, first PDP-11s and later Vaxes.)

Although my history of this is somewhat complicated, I wound up in the Digital 'erase is DEL' camp, partly because it's what at least some of the hardware I used wanted to do and partly because it made life easier in GNU Emacs. Given this, you can probably guess the original cause of my swapping Backspace and Delete in X until recently; I started doing this when my office workstation changed from a DECStation to an SGI Indy. The DECStation's backspace key generated DEL (and all of my environment was set up to deal with that), while the SGI Indy's backspace key wanted to generate Ctrl-H. At the time I made the rational decision that it was simpler to use xmodmap to switch Backspace and Delete than to change my entire environment around.

(I don't know and can't remember what GNU Emacs's state was at the time as far as distinguishing the Backspace key from a typed Ctrl-H. These days they're definitely different under X.)

PS: I don't remember how the original vi behaved with Ctrl-H and DEL, but I think it was friendlier than GNU Emacs as far as dealing with a backspace key that sent Ctrl-H (at least if you had your stty erase character set properly, which you probably did).

Revisiting a bit of my X keymapping history

I started using X what is now a very long time ago, and unlike some people I've never had a complete break with my original X environment, one where I restarted my setup from scratch and threw away all of my old customizations. The natural result of this is that I have been carrying forward some historical decisions without actually ever really looking at them in a modern environment.

At some point in my life with X, one of my customizations became to swap the Backspace and Delete keys via xmodmap. My reason for doing this at the time was straightforward; Backspace was more convenient but generated ^H in various things, while I had set my Unix delete character to ^? (aka DEL) a very long time ago for reasons that seemed to make sense at the time. So things rumbled forwards, and when X programs gave me the choice I set them so that both the Backspace and the Delete keys would generate a DEL or otherwise act the same.

(It's possible that I was actually mistaken about this and my swapping Backspace and Delete was due to a misunderstanding. At any rate, I did it and this is what I can vaguely remember as my reasoning.)

Of course, not all things treat the two keys the same. Many editing fields in X programs use Backspace for 'delete-left' and Delete for 'delete-right', so I got thoroughly acclimatized to reaching off to the far key in order to actually backspace over things in those programs. And there were always a few other anomalies here and there that I just reflexively dealt with.

Recently, for reasons beyond the scope of this entry, I wound up in a situation without my usual Backspace/Delete swap. Much to my surprise, I didn't notice this in xterm, where everything continued just as before (most of how I noticed was realizing that I was deleting characters in a few X programs with the much more convenient Backspace key). While I wasn't paying attention, xterm had quietly decided to start turning both the Backspace and Delete keys into DEL (or at least into whatever your stty erase character is set to; I haven't investigated). Since I have urxvt and Gnome Terminal set up the same way, my xmodmap key swapping turns out to now be both unnecessary and actually a little bit inconvenient.

(Konsole isn't set up to do this, but then I never use it anyways. Sorry, KDE.)

So now I've removed that little bit of xmodmap work from my X dotfiles, and I'm taking a bit of a look at the other keymapping things I'm doing. I can tell you that making CapsLock into an additional Control key is definitely staying, though.

The whole exercise has been interesting and a little bit spooky. I haven't thought about my xmodmap stuff for quite a while now; after all, it worked, right? Yet either it quietly became unnecessary at some point or was never necessary in the first place. I'm sure there's other parts of my X environment that are the same way and I just haven't stumbled over them yet.

(My collection of X resources settings is a good candidate for this. I'm sure there's settings in there for programs that don't even exist any more.)

PS: The widespread use of GNU readline and similar line editing things in programs can make it a little bit hard to see just what characters your Backspace and Delete keys are generating, since by default I believe that readline et al do the same thing with ^H and DEL.

One advantage of System V is that it was available

I mentioned recently that even though I have a negative view of System V, it made its own contributions to Unix. There are a number of technical contributions, but one of the under-appreciated things it did was simply that System V was available.

The reason that Unix vendor after Unix vendor used System V is in large part because AT&T made it available for licensing. My understanding is that it was even licensed on relatively generous terms; it didn't cost particularly much money to get a source license with binary redistribution rights, and AT&T mostly let you do whatever you wanted with the result without many restrictions. The results of this is that from the mid to late 1980s onwards, Unix versions flourished everywhere, from the small to the large. We probably would not have the broad Unix ecology we do today if AT&T had tried to be more restrictive.

This is all the more noteworthy because AT&T itself was in the business of selling Unix machines for much of this time, and they weren't particularly successful machines either. AT&T undercut its own Unix server business by selling AT&T Unix licenses to direct competitors who then generally offered better products with it.

(Although I don't know for sure, I don't believe that AT&T required things like per-system royalties in its commercial Unix licenses.)

Now, I don't know how much money AT&T earned from its Unix licensing business. But either way, AT&T made an unusual decision to let its server hardware business suffer when it might well have been able to give it a hand, a decision that many companies have decided differently. The result of AT&T's decision here drastically helped Unix spread, especially in the basic server market that became the bread and butter of many Unix vendors.

So, regardless of what I feel about System V at a technical level, I have to respect it simply for being available, for being out there in the world and introducing a great many people to Unix.

System V was kind of backwards for a long time

I mentioned recently that I have a blind spot about System III and some cultural biases about System V. Well, calling them 'cultural biases' understates things, because it makes the differences between BSD and System V back then sound like the differences between Linux and the *BSDs today. Unfortunately, the split was nowhere near that nice. In fact, I'm going to be blunt: for most of its life, System V was a fairly backwards system.

Do you like job control in your shell? System V didn't have that. Do you like being in multiple groups at once? Nope. Things like symlinks and renaming directories? Of course not. Paged virtual memory? Believe it or not, System V spent a surprising amount of time without it (although less than I thought until I looked it up). The result was that merely using a System V system at the shell level was fairly different from using a BSD system, in a way that is not the case for Linux versus the *BSDs.

(We won't even talk about TCP/IP, which only showed up in SVR4 in AT&T's releases. And I believe that System V was stuck with 14 character filenames for most of its life.)

While System V had a certain amount of good things, it was a very foreign environment for BSD people to use. Many of us didn't like it very much. Some of the strangest and most foreign feeling Unixes I've ever used have been strongly based on System V (pre R4).

With that said, in practice many vendors started from System V but then folded in any number of BSD features; this was especially popular with people who wanted to sell machines to universities. I understand that the result could be a reasonably decent approximation of BSD.

(I'm sure that people who grew up on System V could write up a similar list of important things that BSD was missing. But I do really believe that System V was behind BSD at a technical level for much of its life, even if it had some good bits. Well. Behind what vendors like Sun and DEC were doing with BSD, at least, as actual BSD Unix basically stopped around the time of 4.3 BSD.)

More on chroot()'s history, and my blind spot about System III

In a comment on my entry on chroot()'s history, Greg A. Woods noted that System III is the first Unix where chroot() actually prevents your process from just doing chdir("/..") to escape the new root directory. System III predates 4.x BSD, so I was more or less wrong in my entry on this. Only in the BSD line was 4.1c the starting point for this bit of chroot() security. System III source code is even available online here, so I could have checked and seen this myself if I'd thought of it.

I didn't, though, and that's because I have a blind spot about System III. For a long time Unix was split into two sides, which I'll call the university side and the commercial side. BSD and all of its descendents come from the university side; System III and then System V came from the commercial side. The university side dominated both in universities themselves and in Sun and DEC workstations that more or less derived from that environment, while the commercial side mostly wound up in high end big iron servers.

(SGI was an odd case; it was System V derived but had a bunch of BSD stuff added. This caused a certain amount of heartburn in people who dealt with it.)

Although I've used System V machines, in cultural terms I come from the university side of Unix; it's what I have the most exposure to, what I'm most familiar with, and as a result it's what I reflexively think of as 'real Unix'. In other words, it's a tribal affiliation. With a few exceptions I tend to just assume that BSD did something first and best, and that System V had a lot of hacks. So when I was looking at the history of chroot(), I didn't pay a lot of attention to System III; I didn't really look to see the state of chroot() in it, and I didn't actually look at its release date (which is surprisingly early).

(It looks like System III and information about it probably wasn't publicly available early enough to influence BSD's chroot() stuff, but it's at least possible I'm wrong here and that hearing about chroot() security in System III helped push BSD to implement it.)

This is, of course, kind of a mistake. System III and later System V had their own innovations, chroot() security among them, and I shouldn't dismiss their contributions to Unix so reflexively and tribally (even if AT&T too often had terrible ideas there).

When chroot() started to confine processes inside the new root

Writing about the somewhat surprising history of chroot() did leave me with one question: when did chroot() start to confine processes inside the new root directory hierarchy? This is an interesting moment because it marks the point where chroot() stops being a little hack to help emulation and instead turns into a security feature.

(The first use of chroot() as a security feature seems to be in the 4.2BSD ftpd, as covered in the first entry. I can't be completely sure of this because I can't find an easily searchable version of the tuhs.org 4.1c BSD tree.)

Early versions of chroot() appear to be trivially escapable by things like 'cd /; cd ..', which puts you in the parent of the nominal root directory. A version of the chroot() system call that did not allow this appears in 4.1c BSD; you can see the code in namei(). Unlike the 4BSD version of the same code, this code specifically checks to see if you are trying to look up '..' at the chroot root directory, and remaps the result if you are.

I don't know for sure why this change appeared in 4.1c BSD, but it's possible to speculate. The 4BSD namei() is essentially the same as the V7 namei(), but the 4.1c BSD namei() is significantly changed in several ways (for example, it has a lot more comments). 4.1c BSD is the first appearance of two significant changes related to namei(); it's when BSD introduced both a rename() system call and the BSD FFS. It also seems to have seen a significant reorganization of the kernel source code away from its previous V7-like appearance. So I suspect that when the BSD people were changing namei() around anyways because of other changes, they noticed and fixed the chroot escape. With the chroot escape fixed, it was then used as a security feature in the 4.2BSD ftpd.

(The history portion of the Wikipedia page on chroot is no help, because it's clearly wrong unless you creatively reinterpret what it's saying. chroot() was not 'added' to BSD at any point, because BSD inherited it from V7 from the start. This bit of history appears to come from the references section of FreeBSD's Jails: Confining the omnipotent root (via) from 2000 and may refer either to the addition of a chroot(2) manpage or the namei() changes.)

Sidebar: The peculiar history of chroot() documentation

In V7, as I discovered, chroot() is documented in the chdir() manpage. However, while 32V, 3BSD, and 4BSD all still have the chroot() system call, documentation for it has disappeared from their chdir() manpages. A chroot() manpage (re)appears only in 4.1c BSD.

The 32V chdir() manpage seems to be the V7 manpage with the chroot() documentation removed (and it definitely isn't the V6 chdir() manpage). It may be that the chroot() stuff was removed because the 32V people thought it was a hack that was better off not being documented, or maybe 32V got their manpages from an earlier version of V7 that didn't have the chroot() addition.

How NFS deals with the pending delete problem

The pending delete problem is that in Unix it's valid to unlink() a file that you (or someone) has open(). If you do this, the processes with the file open must not lose access to it but the file also needs to vanish from the filesystem. If you ignore some issues this is easily handled in the kernel for local filesystems, but when I originally talked about this, I said that Sun had had to come up with a different solution for NFS. So let's talk about that.

The problem with pending deletes on NFS is that NFS is a stateless protocol. The server deliberately doesn't keep track of or even know whether or not clients have a file open; all it sees is a stream of requests for a NFS filehandle. This means that if you tell the server to delete the file, well, it's going to do that; it has no idea whether or not your client still has the file open and expects it to keep working. At the same time the clients can't not make the file go away when they get told to; users and programs that do 'unlink(fname)' are going to get peeved if it fails with 'file is in use' or if it doesn't actually go away.

The solution to this conflict is what sometimes gets called 'NFS silly renames'. When a NFS client is asked to unlink() a NFS file that it knows is still in active use, it doesn't tell the server to delete the file but instead renames it to .nfs<random>. When the last process closes the last file descriptor to the theoretically deleted file, the client kernel finally tells the NFS server to actually delete the lingering .nfs* file. This works surprisingly well and does most of what people expect when they unlink() an actively used file.

(One sign that it works very well is that most people who use NFS have never noticed this going on behind the scenes.)

Of course any number of things can go wrong with this scheme in corner cases (or not so corner cases). The obvious one is that if the client kernel crashes during this process there's nothing left to clean up the .nfs* files. As a result, many NFS servers come with scripts that run find on your filesystems to spot any lingering .nfs* files that are too old and delete them. Another problem is that this only works when everything is on the same client; if you have the file open() on one client and unlink() it on another, well, the second client is just going to tell the server to delete the file and now the first client has a stale filehandle. Such is life with a stateless network filesystem; people have learned to live with it.

(Before people get too down on NFS over this issue, I want to say that in general NFS is a remarkably good and successful Unix network filesystem. That it has minor drawbacks in no way detracts from its major successes.)

The somewhat surprising history of chroot()

The chroot() system call is one of those that I think of as being really old. Up until a while back, if you'd pressed me I'd have guessed that it originated in V7. Then I dug into it and in that tweet claimed it was from 4.2 BSD. This is what you'd kind of expect and what it sort of looks like, but it turns out to be wrong and the history of chroot() seems to be much more interesting than I expected.

The classic use of chroot() is for safely confining network daemons, and indeed this is what ftpd uses it for in 4.2 BSD. But there's another use of it in the 4.2 BSD tree lurking in a file called /usr/src/games/compat/runcompat.c. As a manpage and a Readme there make clear, this is a system to let you run PDP-11 V6 and V7 programs on your VAX. It uses chroot() to optionally put those programs inside a familiar V6 or V7 directory hierarchy. This appears to go more or less all the way back to 32V, the first Unix version to run on a VAX and thus the first one where this desire would come up. But it's not where chroot(2) was added.

Much to my surprise, it turns out that chroot(2) is in V7 after all. It's not obvious because there's no separate manpage for it; instead it's part of the chdir(2) manpage. I don't know why it's there, apart from 'someone thought it was an obvious feature'; tuhs.org has more or less complete V7 source, and there's no programs in the source tree that use it that I can find. Apparently it was added first and then people found various uses for it later. What we now think of as its most prominent uses (for things like confining anonymous ftp) are relatively latecomers to the game, since they started years after chroot() itself came into existence.

(I don't believe that the V7 chroot() was a security measure.)

Sidebar: why chroot is in the chdir manpage

V7 implements chdir(2) and chroot(2) with basically the same code; you can see it in /usr/sys/sys/sys4.c. It's pretty clever:

chdir()
{
    chdirec(&u.u_cdir);
}

chroot()
{
    if (suser())
        chdirec(&u.u_rdir);
}

chdirec(ipp)
register struct inode **ipp;
{
[...]

In other words, the only real difference between chdir and chroot is what field in the u user structure they act on. chdir acts on the 'current directory' field, chroot acts on the 'root directory' field.

The actual implementation of chroot requires a bit more kernel code than this, because of course the u_rdir field didn't exist before chroot was added. But there turn out to be remarkably few places that deal with u_rdir in the V7 kernel. By the way, reading this code makes me suspect that a V7 chroot was remarkably permeable; I don't see any reason why you can't just 'cd ..' your way right out of one (although I may be missing something subtle in the code). That obviously changed later.

(Note that the V7 manpage doesn't claim that chroot() confines processes; it simply says that it changes what '/' means to them. Which is literally what the kernel code does.)

Getting dd's skip and seek straight once and for all

Earlier today I wanted to lightly damage a disk in a test ZFS pool in order to make sure that some of our status monitoring code was working right when ZFS was recovering from checksum failures. The reason I wanted to do light damage is that under normal circumstances, if you do too much damage to a disk, ZFS declares the disk bad and ejects it from your pool entirely; I didn't want this to happen.

So I did something like this:

for i in $(seq 128 256 10240); do
    dd if=/dev/urandom of=<disk> bs=128k count=4 skip=$i
done

The intent was to poke 512 KB of random data into the disk at a number of different places, with the goal of both hopefully overwriting space that was actually in use and not overwriting too much of it. This turned out to actually not do very much and I spent some time scratching my head before the penny dropped.

I've used skip before and honestly, I wasn't thinking clearly here. What I actually wanted to use was seek. The difference is this:

skip skips over initial data in the input, while seek skips over initial data in the output.

(Technically I think skip usually silently consumes the initial input data you asked it to skip over, although dd may try to lseek() on inputs that seem to support it. seek definitely must lseek() and dd will error out if you ask it to seek on something that doesn't support lseek(), like a pipe.)

What I was really doing with my dd command was throwing away increasing amounts of data from /dev/urandom and then repeatedly writing 512 KB (of random data) over the start of the disk. This was nowhere near what I intended and certainly didn't have the effects on ZFS that I wanted.

I guess the way for me to remember this is 'skip initial data from the input, seek over space in the output'. Hopefully it will stick after this experience in toe stubbing.

Sidebar: the other thing I initially did wrong

The test pool was full of test files, which I had created by copying /dev/zero into files. My initial dd was also using /dev/zero to overwrite disk blocks. It struck me that I was likely to be mostly overwriting file data blocks full of zeroes with more zeroes, which probably wasn't going to cause checksum failures.