Wandering Thoughts archives

The ecological niches of current open source Unixes

There are a fair number of open source Unixes around and over time I've wound up with some jaundiced perceptions of what their various ecological niches are. Today I feel like writing them up, with the disclaimer that this is not just in my own view but that it is my view of what perceptions of these Unixes are, not necessarily the reality.

Linux

Everyone knows this one. Linux's niche is as the popular and widely used open source Unix, and as a result it's the one that almost everyone makes sure their programs run on and writes software to. On its primary hardware platforms, Linux's hardware support is basically unmatched; if there's a driver for something, Linux was probably the first target. Linux is what you use by default or if you don't want to take any risks.

(For example, notice that mozilla.org provides Firefox builds for Linux but not for other open source Unixes.)

FreeBSD

FreeBSD's niche is being the non-Linux. It's what you run if you want a well done free Unix that is not Linux. It doesn't get the love (and the software) that Linux does, but it's popular enough to mostly keep up in most settings. The developers of open source programs (at least ones that aren't intended to be Linux specific) will probably not laugh at you if you report 'fails to work properly on FreeBSD' bugs to them.

OpenBSD

OpenBSD has determinedly created a niche as the firewall and networking Unix and as what you run if you care much more about (remote) security than anything else. Note that this is not quite the same as 'the secure one'. Because of its focus on security and the accompanying minimalism OpenBSD is generally behind the times in terms of features, software versions, and so on (the abrasive personalities involved don't help either). The developers of most open source programs may just look at you bemusedly if you report 'fails to work properly on OpenBSD' bugs.

NetBSD

To the extent that this has an ecological niche, it's running on CPU architectures and hardware platforms that are too obscure or old for Linux (or anything else). The motto on their website really summarizes it well: "Of course it runs NetBSD". The flipside of this is that NetBSD is not something you run if you have a choice.

The remaining open source Unixes are either effectively in very specialized niches (regardless of what they wish to be seen as), lack clear niches where they are the leading choice, or both. For example, DragonFlyBSD would probably like to be in FreeBSD's ecological niche but is too obscure.

(You may also want to see my entry on where the Unix community wound up.)

Why the defaults for tunable kernel parameters matter

Yesterday I ranted in passing, as I have before, about Unix vendors that have never adjusted the defaults for any tunable kernel parameters and thus leave parameters set at values that were perhaps correct fifteen years ago on much different machines, but which certainly aren't any more. Given that tunable kernel parameters can after all be changed to better values, it may not be obvious why this is a problem.

There's two reasons why you should get tunable parameters correct from the start. The first is that people will get artificially and unnecessarily bad performance until they learn enough to change the parameters (if they ever do). I suspect that many people never change the defaults and instead go along either vaguely wondering why their system performs badly or simply thinking that this is the intrinsic speed of your system. If this seems irrational, well, note that there is generally a large collection of tunable parameters, many of them are badly documented, and then it's far from obvious which parameters (if any) are applicable to your problem. Especially if you don't have detailed knowledge of the kernel.

This leads to the second problem, which is that the OS vendor's kernel people will normally know a lot more about the meaning, effects, and sensible values of kernel parameters than some random sysadmin will. Having the system tuned by the latter instead of the former is not something that makes much sense; the former is going to do a better job (and sometimes a significantly better one). This is not a new or novel problem; instead it's the same old problem you have any time the experts punt hard issues to non-experts instead of dealing with them.

(In fact many sysadmins are going to use whatever settings a quick web search leads them to. This is sort of okay if the top hits of such searches have sensible advice, but you lose badly if the top hits lead people to bad advice.)

Why System V shared memory may normally be limited

I recently read Robert Haas's Absurd Shared Memory Limits, in which he writes:

Despite the fact that System V shared memory is the only available shared memory implementation that provides this feature, most operating system vendors frown on it, and encourage users to use the newer POSIX shared memory facilities instead. They do this by limiting the amount of System V shared memory that can be allocated by default to absurdly small values, often 32MB.

To put it one way, I'm not sure that Unix vendors are being quite so calculating and deliberate here. System V shared memory has had low limits for basically as long as it's existed, so at a minimum OS vendors are just leaving things untouched through neglect (at the maximum, they're deliberately choosing not to increase the defaults to sane levels). There's a lot of this neglect in the kernel parameters for current Unixes, especially ones with a long history.

(Old time Unix vendors sometimes seem to act as if their customers would freak out if the vendors changed any tunable kernel parameters. This leads to absurd results given that memory, CPU speeds, and the number of CPUs in a system have all increased drastically over just the past decade. I've written about this before and it's somewhat of a hot button for me.)

However, my fallible memory suggests that there may have been another reason behind those low numbers, at least back in the day. Put simply, I believe that System V shared memory was initially not swappable. This would have made it doubly dangerous; it could be used by anyone, not just privileged processes, and these memory segments stayed around even after processes exited. In that situation, a quite low default limit makes a lot of sense.

(Of course, System V SHM is swappable on current systems and has been for years. If the low limits were for this reason, the reason's been inapplicable for, oh, a decade or two. But then we have that whole fossilization of tunable kernel parameters thing.)

Now, the disclaimer: I'm not sure that this was actually the case, and I can't find any sources online that would tell me either way. I may well be misremembering.

PS: even if current Unix vendors are leaving the System V SHM limits unchanged from the days of much smaller systems partly through a desire to have everyone move to Posix SHM, I suspect that this is more likely to be a policy of deliberate neglect than a policy of deliberately frustrating people.

Sidebar: how System V SHM could easily have been unswappable

In a modern Unix kernel design, leaving System V SHM unswappable would be a vaguely bizarre behavior. While SysV SHM is not directly tied to a process, there's any number of similar oddities that do get swapped and there's various bits of infrastructure that can be hijacked to help out.

However, System V SHM did not originate on modern Unix kernels; it originated on kernels that were so old and basic that they did not even have paging (just whole-process swapping). In this environment, swapping System V SHM segments would have required an entire new chunk of infrastructure, and all of the System V IPC mechanisms were basically quick hacks. I find it all too plausible that the initial implementation of System V SHM would simply have not bothered to build the extra kernel bits necessary to swap SHM segments.

(I do not have a high opinion of System V IPC in general, but that's for another entry.)

The well behaved Unix program and job control signals

The TTY demystified (which I've commented on before) has some things to say about the SIGTT* job control signals. In the process it says something about how they are and should be handled that I think is dangerously misleading.

Specifically, contrary to what The TTY demystified says, you shouldn't assume that the shell saves and restores TTY modes for you when your program is suspended and restored. A well behaved Unix program that changes the TTY modes will remember and restore the original modes when it gets SIGTTIN or SIGTTOU, and then re-set them to what it needs when it gets SIGCONT. Note that a well behaved program may need to do much more than just save and restore TTY modes; one of the reasons that programs get sent the SIGTT* signals (instead of just being abruptly suspended) is so that they can do things like reset the terminal state away from whatever peculiar colours and other settings they've placed it into.

(This is where the alternate screen plague kicks in.)

These days I think that basically all surviving shells (at least ones with job control) do save and restore TTY modes for you, because sadly this is basic self defense on part of the shell. However, they generally don't reset the terminal state (partly because they often have no idea what state you've put it into).

(This has implications for how your program should handle SIGTTIN, but that'll take another entry.)

The article also asks (after discussing ^S/^Q flow control):

I don't know why the designers of UNIX had to go all the way to invent SIGTTOU and SIGTTIN instead of relying on blocking I/O, but my best guess is that the TTY driver, being in charge of job control, was designed to monitor and manipulate whole jobs; never the individual processes within them.

The straightforward answer is that the SIGTT* signals are designed to give your program a chance to react to being suspended. That's why you can catch them at all, unlike SIGSTOP. If the TTY driver simply blocked your process from doing further TTY IO (or just sent you a SIGSTOP), your program couldn't clean up any special state it had set and doing this would be entirely on the shoulders of the shell (which cannot do it as well as your program can in many situations).

Sidebar: why the shell has trouble restoring the terminal state

At one level and in theory, restoring the terminal state is simple; the shell can just send the terminfo state reset sequence. However there are two problems with this. The first is the general problem of interrupting an operation in the middle; if the program was abruptly suspended partway through sending the terminal an escape sequence, the terminal may be in a mode where it will interpret part of your reset sequence as part of the incomplete escape sequence.

The second problem is that sometimes, part of 'resetting' things is program dependent. The big example is where you should (re)position the cursor. The cursor may have been put in an arbitrary and potentially strange position and thus needs to be repositioned somewhere useful, or it might be in a perfectly sensible location already and you should leave it alone. The first case is typical of a full screen program like vi (where the cursor will usually be repositioned to the bottom of the screen); the second case is typical of a program that just uses readline. The shell generally can't know which program is which.