Wandering Thoughts archives

An index of non-letter control characters

In Unix and ASCII, bytes 1 through 26 are control-A through control-Z. But there are control characters outside that range: 0, 27 through 31, and 127. As a result of writing up my _.screenrc I've become curious about what all of them are, so here's my handy chart:

The canonical representation of a control character is based on what Unix prints it as when you enter it (usually with Ctrl-V followed by the character, however you generated it). It turns out that xterm will generate a number of these sequences with alternate Ctrl combinations as noted in the chart (which is probably not complete). Some of these xterm alternates may be more convenient under some circumstances.

Your mileage on actual serial terminals and other terminal emulators may vary, although gnome-terminal and urxvt match up for the primary control sequences and at least some of xterm's alternate ways of generating them. Historically serial terminals could be very variable outside Ctrl-A through Ctrl-Z.

Backspace traditionally sends Ctrl-H. Ctrl-\ is traditionally what stty will report as 'quit', ie it sends SIGQUIT ('please die down with a core dump') to programs; this can make it kind of hard to work out just what ASCII code it represents. I resorted to entering it in vi, saving the file, and then using od to dump the file.

In case you ever need to know this, Ctrl-J is the real end of line character in terminal entry, aka \n; the tty driver maps Return (aka Ctrl-M aka \r) to it normally, but this can be disabled and then you can be stuck if, eg, you crashed a program that uses 'raw mode' when you were trying to work out the ASCII number for Ctrl-\. Many but not all shells will accept Return (or a literal Ctrl-M) as a synonym even in this situation, so you can type commands, but any actual command that prompts you for something probably won't.

(The specific tty mode flag this is controlled by is ICRNL, per CBreakAndRaw. It follows that 'stty icrnl' will help restore a broken session, although you might as well go all the way to 'stty sane'.)

Sidebar: the odd case of konsole

Konsole works just like xterm with the sole exception that Ctrl-? does not seem to generate a Ctrl-? (it can be generated with Ctrl-8, though). Instead konsole appears to swallow Ctrl-? outright, which may have something to do with some sort of magic DEL handling it's doing.

An init system has two jobs

Ever since System V init came on the scene, it's been clear that a Unix init system really has not one but two jobs. I don't quite mean this in the sense of init's historical roles but in a more high level sense of what people want from their init.

The first job is to be init, PID 1, with everything that that implies. Init starts things on boot, supervises processes, does things on shutdown, doesn't crash, and so on. This job is well recognized and understood and as a result, essentially any real init system is pretty good at it (although some are better around the edges than others). Certainly any of them will get the job done.

The second job of an init system is to help you manage the init system and the services and processes it controls. This is the job not of starting and stopping things but of letting you control what is started and stopped and so on. Not being an init system, but controlling the init system. This job is much less well recognized and as a result much more immature (partly because the job only really started when System V init showed that perhaps there might be a better way to do it than editing /etc/rc.local and running by now). daemons by hand, and even then it stagnated for quite a while).

Init systems are all over the map when it comes to the second job. Some of them still barely acknowledge that it exists; some sort of handle it; some are actually pretty decent. A few are genuinely pleasant. Some, unfortunately, are wretched at it. In short, this is the area where init systems really distinguish themselves from each other.

(There are differences in how init systems approach the first job and some of them do make a difference, but my view is that it's nowhere near as big a difference as the second job creates.)

As a system administrator I find this kind of an unfortunate state of affairs, since that second job is something we get to deal with on a regular basis. I really wish most init systems paid more attention to it.

The fundamental problem that created su

There is a fundamental problem that su, sudo, and many other tools are used to try to solve. Actually there are two problems, but su only addresses the one so let's start with it.

The first problem is that you have the right to exercise all power but you do not normally want to exercise that power. Most of the time you want to have no power and perhaps some of the time you might like to have some focused power; this is primarily to limit the scope of accidents and mistakes. If you are duly authorized to do everything on a Unix system, well, perhaps you could just give your account a UID of 0 but almost everyone agrees that that would be a terrible idea. It is much safer not to have all power almost all of the time and to exercise that power for as short as possible a time and in as limited as possible a way.

(Well, really it is 'feasible' instead of 'possible'. There is a balance of tedium between free exercise of power and significant amounts of work.)

Or in short: su is there to limit a sysadmin's power, not increase it. Su is there to let you do things without the powers of root. Because su can give you power temporarily, you do not need that power permanently.

(I suppose technically you could say that su is there to increase auditability, since you can su to root and thereby leave a trace of who you used to be instead of having to log in as root in some way.)

The other problem is that you have the rights to some powers but not other powers and you must somehow be granted selective access to only those powers you have a right to. This problem is solved by setuid on selected programs, sudo in various forms of usage, and with things like RBAC. Su does not solve it directly.

These two problems are almost completely orthogonal. A solution for access to limited powers for people authorized for them does not really help people who have the right to all power but do not want any power most of the time. Perhaps in the right situation it can make their life somewhat safer if people (or scripts) diligently remember what operation needs what limited power and that limited power can be kept turned off most of the time.

(And a solution to access to limited powers does not necessarily mean that those limited powers can be turned off when you don't want them. They may be always-on enhancements to an otherwise normal account.)

This is why RBAC or an equivalent is not a replacement for su for fully root-authorized sysadmins. In fact in some ways bare RBAC is an anti-replacement, if your ordinary account winds up with more intrinsic powers than before.

This is a generic problem for any system of enhanced powers. Unless the additional powers are harmless, the sensible default state is that you do not have the powers despite having the right to them; you want an explicit activation step for safety (and you may want this activation step to involve an additional challenge or secret for increased security). Su (and traditional UID 0 in Unix in general) is only unusual in that the powers are an all or nothing affair instead of selective.

(There are relatively few additional powers that are genuinely harmless. Powers that enable system changes are obviously dangerous but even mere information disclosure powers may give you normally private information that you don't want to know or should not know without real need.)

Vi's composability antecedent (or one of them)

These days the venerable vi editor is getting a bunch of praise for its genuine and core innovation of composable commands, such as in Mike Kozlowski's Why Atom Can't Replace Vim. I wholeheartedly support this; vi's command composability is amazing and powerful and I wish more programs used or even embraced the ideas behind it (as far as I know, vi's basically the only editor that's ever really done this).

But today I want to mention that vi's composability is not without antecedents, in fact antecedents in Unix itself. The antecedents are nowhere near as fully developed and nice as vi, but in them I think we can see the kernel of vi's idea (as well as some ideas that did not catch on in vi). In fact, one antecedent is even buried in vi itself. I am of course talking about the family of Unix line oriented editors that gave us ed, sed, and vi's ex mode.

Like vi itself, all of these editors revolve around a fundamental idea: you specify first text addressing and then the command to apply to whatever you've specified. Want to delete the next five lines? First you specify five lines as '.,.+5', and then you say to 'd' what you've just addressed. Learn a new addressing method and you can use all of the commands you know with it; learn a new command it works with all of the addressing methods you know. This is the same core engine as vi has, although in a much more primitive and limited form of both addressing methods and available commands.

(And if you want to see things that way you can think of Unix pipelines as a form of this idea. Clearly pipelines compose things together, although in a more general way than vi et al do.)

PS: to note this antecedent is in no way to take away vi's striking accomplishment in generalizing it and applying it to more sophisticated movement and addressing as well as more complex text manipulation, and also figuring out how it should all work in a visual, character based environment instead of the basic line oriented one that ed uses.

(See also the history of ed, which stretches back even before Unix.)

The state of limits on how many groups you can be in (especially for NFS)

It is an article of vague Unix folklore that your users can only be in so many groups before they may start having weird problems or things just fail outright. Depending on who you ask and what conditions you're operating under, this limit might be 16 groups, 32 groups, or a lot more. There are actually two limits: the local system limit and a limit imposed by many NFS configurations.

On modern Unixes that you'll want to use, the local limit is generally high. Based on reading manpages and include headers, on current Linux machines it's 64K groups and on current FreeBSD it's 1023 groups. OpenBSD appears to be quite old fashioned with a 16-group restriction. Current Illumos source defaults to a 16-group limit but this can be tuned in /etc/system with 'set ngroups_max=...' if I'm reading the source code correctly.

(See also this resource for other systems and older releases.)

But all of this local optimism is moot in the face of a long standing NFS restriction. Commonly used parts of the NFS protocol are restricted to using 16 groups at most. This limit appeared in the original NFS but it has not been raised in NFS v3 or even, sort of, in NFS v4. More specifically, this limitation is part of the default 'plain Unix permissions' authentication, called AUTH_SYS, that is normally used in NFS v3 and may still be used in NFS v4. In theory you can switch NFS v3 to use something else, but in practice Mike Eisler notes that NLM locking may still cause you problems (that article is a good background in general and discusses potential workarounds).

If you have Linux NFS servers there is a potential workaround where you can make the server ignore the list of groups that the clients send and look up group information locally. There is still a size limit but it's much larger. See Kyle Anderson's Solving the NFS 16-Group Limit Problem for the details. I haven't tested this, partly because our NFS servers don't run Linux so I can't use it.

So, in short, we're going to keep bumping into the 16-group limit every so often for some people here and there is no good workaround for us.

(Every so often I set out to investigate something and wind up finding out that there is no good news and things are exactly as my vague memory of folklore thought they were. Today is one of those cases.)

The security model of sudo versus su

Some time ago I wrote about the three faces of sudo, where one of the faces was basically 'sudo as the replacement for su in how sysadmins get unrestricted root privileges' and I said that this created a different security model than with su. Today I want to talk about what that new model is and how I see it differing from su's in a way that makes me like it less.

To start with we need to talk about the security model of su. Su has a simple security model: you must know the root password and generally be in a restricted group. Then you can either get a root shell or execute a command as root by giving su the root password, and you must do this for every shell or command that you want.

(If your version of su is not configured to require membership in group 'wheel' or the local equivalent, the security model is that anyone at all who knows the root password can use su to assume root powers.)

Sudo's normal security model is that if you are in a restricted group and know your own password, you can run commands as root or get a root shell by typing your own password. Once you've typed your password once you can then run commands as root without any prompt for a certain amount of time.

There are two big differences between these two security models. The first is the choice of password that you are challenged with, where sudo's choice is noticeably weaker. With sudo the security of root access rests not on the root password alone but the passwords of everyone who has root access. Since people's regular passwords are used much more often and more broadly than the root password, my view is that they are more exposed (including stored in programs). An attacker who compromises even one sysadmin password now has immediate access to root by logging in as the sysadmin and repeating the password that they already know; they don't need to compromise an additional secret.

The second is that sudo effectively creates a temporary environment where you have root powers by using a magic prefix. An attacker that can detect this environment (or try blindly) and inject commands into it (either directly or, for example, via a shell script that you're running) can hijack this special power even without knowing your password and immediately use root powers. My view is that this makes a post-sudo shell session unusually dangerous and damaging. It also somewhat increases the possibility of simple accidents, since an errant sudo command will go through without a pause that would give you a chance to reconsider.

Either or both of these changes can be turned off via sudoers settings. But then you've reduced sudo to a variant of su. Maybe that's what you want (sudo does have better logging than su and some additional features), but I don't think it's the common way to use sudo as a su replacement. Maybe I'm wrong here, though; I'd be interested to hear if places that use sudo as a su replacement turn off these.

A Unix semantics issue if your filesystem can snapshot arbitrary directories

A commentator on my entry on where I think btrfs went wrong mentioned that HammerFS allows snapshots of arbitrary directories instead of requiring you to plan ahead and create subvolumes or sub-filesystems as btrfs and ZFS do. As it happens, allowing this raises a little question about Unix semantics. Let's illustrate it with some hypothetical commands:

mkdir a b
touch a/fred
ln a/fred b/
snapshot --readonly a@test
ls -li a/fred .snap/a@test/fred
rm -f b/fred
ls -li a/fred .snap/a@test/fred

Here is the question: what are the inode numbers and link counts shown for .snap/a@test/fred in the two ls's?

Clearly the real a/fred retains the same inode number and goes from a link count of 2 to a link count of 1 after b/fred is removed. If the snapshotted version also changes link count what we have is observable changes in filesystem state in a theoretically read-only snapshot, and also it's not clear how to actually implement this. If the link count doesn't change it's actually incorrect since you can't find anywhere the theoretical second link for .snap/a@test/fred. But at least it's easy to implement.

(As far as filesystem boundaries go, the only sane choice is to have .snap/a@test be a separate filesystem with a separate 'device' and so on. That avoids massive problems around inode numbers.)

All of this is avoided if you force a to be a separate filesystem because then you can't create this link in the first place. You're guaranteed that all hard links in the snapshot are to things that are also in the snapshot and so won't change.

In practice most people probably don't care about accurate link counts for files in a snapshot and this is likely just a nit. But I can't help but come up with these odd corner cases when I think about things like snapshotting arbitrary directories.

(I suspect this hard link issue is one reason that ZFS doesn't allow you to turn an existing directory hierarchy into a sub-filesystem, a restriction that sometimes annoys me.)

Cross-system NFS locking and unlocking is not necessarily fast

If you're faced with a problem of coordinating reads and writes on an NFS filesystem between several machines, you may be tempted to use NFS locking to communicate between process A (on machine 1) and process B (on machine 2). The attraction of this is that all they have to do is contend for a write lock on a particular file; you don't have to write network communication code and then configure A and B to find each other.

The good news is that this works, in that cross system NFS locking and unlocking actually works right (at least most of the time). The bad news is that this doesn't necessarily work fast. In practice, it can take a fairly significant amount of time for process B on machine 2 to find out that process A on machine 1 has unlocked the coordination file, time that can be measured in tens of seconds. In short, NFS locking works but it can require patience and this makes it not necessarily the best option in cases like this.

(The corollary of this is that when you're testing this part of NFS locking to see if it actually works you need to wait for quite a while before declaring things a failure. Based on my experiences I'd wait at least a minute before declaring an NFS lock to be 'stuck'. Implications for impatient programs with lock timeouts are left as an exercise for the reader.)

I don't know if acquiring an NFS lock on a file after a delay normally causes your machine's kernel to flush cached information about the file. In an ideal world it would, but NFS implementations are often not ideal worlds and the NFS locking protocol is a sidecar thing that's not necessarily closely integrated with the NFS client. Certainly I wouldn't count on NFS locking to flush cached information on, say, the directory that the locked file is in.

In short: you want to test this stuff if you need it.

PS: Possibly this is obvious but when I started testing NFS locking to make sure it worked in our environment I was a little bit surprised by how slow it could be in cross-client cases.

Partly getting around NFS's concurrent write problem

In a comment on my entry about NFS's problem with concurrent writes, a commentator asked this very good question:

So if A writes a file to an NFS directory and B needs to read it "immediately" as the file appears, is the only workaround to use low values of actimeo? Or should A and B be communicating directly with some simple mechanism instead of setting, say, actimeo=1?

(Let's assume that we've got 'close to open' consistency to start with, where A fully writes the file before B processes it.)

If I was faced with this problem and I had a free hand with A and B, I would make A create the file with some non-repeating name and then send an explicit message to B with 'look at file <X>' (using eg a TCP connection between the two). A should probably fsync() the file before it sends this message to make sure that the file's on the server. The goal of this approach is to avoid B's kernel having any cached information about whether or not file <X> might exist (or what the contents of the directory are). With no cached information, B's kernel must go ask the NFS fileserver and thus get accurate information back. I'd want to test this with my actual NFS server and client just to be sure (actual NFS implementations can be endlessly crazy) but I'd expect it to work reliably.

Note that it's important to not reuse filenames. If A ever reuses a filename, B's kernel may have stale information about the old version of the file cached; at the best this will get B a stale filehandle error and at the worst B will read old information from the old version of the file.

If you can't communicate between A and B directly and B operates by scanning the directory to look for new files, you have a moderate caching problem. B's kernel will normally cache information about the contents of the directory for a while and this caching can delay B noticing that there is a new file in the directory. Your only option is to force B's kernel to cache as little as possible. Note that if B is scanning it will presumably only be scanning, say, once a second and so there's always going to be at least a little processing lag (and this processing lag would happen even if A and B were on the same machine); if you really want immediately, you need A to explicitly poke B in some way no matter what.

(I don't think it matters what A's kernel caches about the directory, unless there's communication that runs the other way such as B removing files when it's done with them and A needing to know about this.)

Disclaimer: this is partly theoretical because I've never been trapped in this situation myself. The closest I've come is safely updating files that are read over NFS. See also.

Rebooting the system if init dies is a hack

I feel like I should say this explicitly: rebooting the system if init dies is a hack. It's the easy thing to do but not the right thing. V7 Unix more or less ignored the possibility of init failing; when BSD started considering this situation they took the easy way out of 'handling' it by just rebooting the system. Everyone since then has copied BSD (probably partly out of compatibility, since 'everyone knows' that if init dies the system reboots, and partly because it's the easy way).

You can argue that if init dies something terrible is going on (especially after the kernel has armored init so that you have to work very hard to terminate it) and this is generally true. But rebooting the system is the lazy way out, especially when this is determined by the kernel instead of user level. It might certainly be sensible to configure your system to immediately start a reboot if init ever dies and is restarted by the kernel, but at that point it's something you control at user level; you might instead ring lots of alarms and see if the system could limp on. And so on. From some perspectives, 'reboot the system if init dies' is the kernel meddling in policy that should be left to other levels.

The right thing is to provide some way to recover from this situation. I outlined two plausible approaches yesterday; there are probably more. Of course this is more work to design and program than just rebooting the machine, but that's common when you do the right thing instead of the easy thing.

It's kind of sad that almost everyone since BSD has simply followed or copied the BSD quick hack approach (even the people who reimplement things from scratch, like Linux) but this is pretty typical for Unix. If some Unix did try to do it differently I suspect that there would be people complaining that that Unix was over-complicating init.