Wandering Thoughts archives

Why the kernel does mmap() (beyond efficiency)

I recently read Daniel Ehrenberg's Why not mmap? and ran across the following snippet:

mmap() is a system call, implemented by the kernel. Why? As far as I can tell, what I described above could be implemented in user-space: user-space has page fault handlers and file read/write operations.

One way to put the reason that the kernel handles mmap() is that there is a good argument that the reason mmap() exists is shared libraries. And shared is the operative word here. A kernel implementation of mmap() makes it easy to share mapped objects between different processes, in several different ways.

First and most obviously, the kernel can simply share the memory if several processes mmap() the same thing. Such as, for example, much of the C shared library. Second, the kernel can share memory using copy on write semantics; if you map an object 'private', the kernel can still share pages that you don't actually write to. This is commonly used for making necessary per-process modifications to mapped shared libraries, such as relocations or read-mostly data. Third, the kernel can have a unified buffer cache so that it doesn't matter whether processes use mmap() or read() and write(), they all see the same thing and the kernel doesn't duplicate memory.

It might be possible to emulate the first sort of sharing at user level between cooperating processes (you'd use System V shared memory). I don't think that the second sort of sharing is doable at user level unless you have new address space modification syscalls that allow you to overlay chunks of regular address space on System V shared memory (you'd normally do something like that with mmap() and friends, but we're assuming that they don't exist). The third sort of sharing is impossible without kernel assistance, and I don't think one should under-estimate how important it is; read() vs mmap() coherence is what makes it relatively trivial to replace read() with mmap() in programs.

The first and the second sort of sharing are both very important to get full benefit from shared libraries. If you simply load shared libraries into processes without sharing, all you save is disk space; if you share the memory of shared libraries between processes, you also save precious RAM. Back in the days of SunOS 4, with small memory machines and large shared libraries for things like graphics toolkits, this made a real difference.

(People did shared libraries with special custom hacks before there was mmap(), but mmap() had a number of advantages including being clearly generic and providing a clean answer for how to deal with different processes loading the same shared library at different addresses in memory.)

X and the misleading claim of 'mechanisms not policy'

One of the mantras of X has always been that it provides mechanisms, not policy; the policy is up to whatever you run on top of it. So X does not dictate a specific look and feel, or even how windows are managed, and boosters of X are quite proud of this. Various X programs have adopted some close variant of this mantra for their own guiding principle; for example, I believe that a fair number of window managers take this approach (especially fvwm-style window managers). The problem is that this has always been an exaggeration. X is not as policy free as it likes to pretend, because the choices of what mechanisms it offers constrain your choice of policies. As an example, let's consider window placement.

In theory, X is policy-free on window placement; your window manager can position new windows however it wants. It can tile them, it can automatically place them in some optimal way (as many modern user-friendly window managers do), it can let people choose their locations in any number of ways, and so on. All of this works. Now, let us consider another way to handle placing new windows: you pick a menu item, for example 'new terminal', immediately sweep out where you want the new terminal window to go, and then it appears there.

(Let us call this the Blit method of window placement.)

You cannot implement this window placement policy in X, at least not in a way that lets you work with existing X clients. This is because the mechanisms that X supports for window placement are all asynchronous methods; they contain the fundamental assumption that creation of new windows is outside the control of the window manager. New windows just show up and the window manager has to deal with it. There is no mechanism to create a window and hand it to a program to use, nor is there a way to link an earlier window manager action to a new window from a new program (so your window manager cannot have you sweep out the location of the new window, remember that it is for terminal window such and such, start the terminal program, and when the program creates that window the window manager already knows where it should go).

Similar limitations on the possible policies can often be found in X programs that have adopted this mantra. For example, I suspect that many 'mechanisms not policy' window managers could not be used to create a tiled-windows environment; the mechanisms that you'd need for such a thing fall outside the mechanisms in the construction kit provided by the window manager.

(Sometimes you can fake it and it will mostly work, just as you can sort of fake the Blit method of window placement under limited circumstances.)

Fvwm as a window manager construction kit

I've mentioned in the past that I use fvwm as my window manager. However, that doesn't tell you as much as you might think about what my environment looks like because fvwm is not so much a window manager as a window manager construction kit, in much the same way that exim is not a mailer but a mailer construction kit.

Historically (and currently) there have been (at least) two sorts of window managers for X. The first sort is more or less fixed window managers such as Metacity, where most of their behavior and look is hardcoded and can at most be customized somewhat through various settings and options. The other is what I'm calling window manager construction kits, where you don't so much customize them as create a complete configuration and where much of the window manager's behavior is changeable.

(Note that both sorts of window manager can have a good 'out of the box' experience, without you configuring things. The difference is that with a window manager construction kit this happens because someone wrote a good default configuration file; with a window manager, the experience is just how the window manager works.)

Programmers like window manager construction kits for all of the obvious reasons; it's like coding and we generally have a burning urge to customize our environment. Normal people generally hate them, partly because very few normal people want to choose options and customizations by learning a new language and a complex environment and then editing a file that can totally destroy your X session if you get things wrong.

I like fvwm and I'm pleased that it's finally officially released a stable version (2.6 was 'in development' for years and the 2.5.x development series long since converged to stability). But I don't fool myself that this news is of any general interest.

PS: of course there are intermediate stages between being a fully fixed window manager and being a pile of construction pieces. There are any number of window managers that have fixed ideas about some window management issues but have you construct everything else; I believe that some number of tiling window managers fall into this category.

The importance of xterm for X Windows

In an earlier entry I said that you could make a strong case that X succeeded because its authors spent so much effort on making xterm a good terminal emulator. Today I feel like explaining that.

First, let's be clear that xterm was (and is) a very good terminal program. It was fast and responsive, it worked well, it emulated almost everything that a very popular real terminal did (which was important because at least some things basically assumed that they were talking to a vt100-compatible terminal), and it had excellent cut and paste support for straightforward text. These were not necessarily common attributes in terminal programs at the time (and even later; NeWS had a famously slow terminal emulator if I remember correctly).

(People sometimes knock xterm's cut and paste support as being non-standard. That is true, but its cut and paste support is also extremely fast and easy to use; if you are going to use it a lot, this matters much more than having a slower but more standards compliant version.)

This mattered because almost everything people did in new Unix windowing systems at the time was, well, run terminal windows. This is because the new windowing system wasn't going to start out with very many desirable graphical programs for it (and X certainly didn't); as a result, most of what people did with it was run their existing terminal-based programs in terminal windows. How nice those terminal windows were thus made a big difference to how enthused people were about the overall windowing system; a bad terminal program made the whole system unpleasant or annoying to use, while a good one made it a pleasure.

(The web has changed this drastically; these days the majority use of any graphical system is probably to run a browser. But this was back in the mid to late 1980s and early 1990s.)

My strong impression is that a lot of the competing window systems considered their terminal program to basically be an afterthought (which resulted in the sort of terminal program that you'd expect); most of their focus was on building a graphical toolkit and then some number of graphical programs to alternately show it off or do useful work. The problem was that of course these systems could never build enough graphical programs to make the terminal program unnecessary, even if Unix people at the time had been willing to give up the command line. The result was that in practice the experience of using those systems tended to be kind of unpleasant unless you were doing something that their particular set of GUI programs were good at.