Wandering Thoughts archives

Why accurately counting committed address space is hard

One of the problems with the whole virtual memory overcommit issue is the difficulty of working out how much memory truly is committed. For example, consider all of the ways that Unix kernels can commit themselves to holding down memory on behalf of a user process. In addition to obvious things like System V shared memory, this includes (but isn't limited to):

• pipe data buffers, including unused ones; theoretically every pipe pair created commits the kernel to at least 4K of buffer or so.
• socket send and receive buffers, which are often quite large in theory but very small in practice.
• pending writes that have not been flushed to 'disk'.

You can, with sufficient work, make the kernel count all of these so that it knows exactly how much memory has been committed. Then you run into two additional complications in practice: first, that much of this committed memory won't necessarily every get used (especially for socket buffers), and second that the kernel itself will need a certain amount of additional memory to process these things.

(The latter matters because the real goal with committed memory accounting is to make sure that the system never deadlocks or has to start killing things in order to make forward progress, and for that you need to include the kernel overhead. The former matters because people don't want to be restricted unless the system really is out of memory in some way.)

The good news is that many of these are just using memory; there are fewer that are committing memory that is not yet there (pipe buffers, for example, since POSIX specifies that you can write a certain amount of data to a pipe without blocking).

A bash irritation: mutable history

Here's one of those little things that irritate me about bash, or more specifically its command line editing: its command history is (sometimes) mutable. Under at least some circumstances, you can overwrite a command that is in your history by going back to it (with search or cursor up) and editing it again. When you do this, the old version of the command is gone permanently; if you cursor back to it again, you see your freshly edited version.

(This doesn't seem to happen all of the time or on all of the machines I use bash on. So far I haven't had the patience to track down exactly what's going on.)

My mental model of shell history is that the past is immutable because it is, after all, the past; editing a past command should just result in a new command that just looks a lot like the old one. To have a sometimes mutable command history makes me nervous and twitchy; among other things, that old command may be the only version where I had something working (and I am editing it to try to improve things, which may or may not be successful).

I do have some sympathies with the mutable command history crowd. If you think of the command history as basically an edit buffer, it's more natural to change things in place, and even if you don't there are a bunch of interesting decisions (and painful programming) to do to make things come out right.

(Thinking about some torture test scenarios, I think you need to pretend that you have a mutable edit buffer and then undo all of the mutations at certain trigger events and insert any new command line at the end of the history. I maintain that one such trigger event should be clearing the entire line, however it happens.)