Why accurately counting committed address space is hard
One of the problems with the whole virtual memory overcommit issue is the difficulty of working out how much memory truly is committed. For example, consider all of the ways that Unix kernels can commit themselves to holding down memory on behalf of a user process. In addition to obvious things like System V shared memory, this includes (but isn't limited to):
- pipe data buffers, including unused ones; theoretically every pipe pair created commits the kernel to at least 4K of buffer or so.
- socket send and receive buffers, which are often quite large in theory but very small in practice.
- pending writes that have not been flushed to 'disk'.
You can, with sufficient work, make the kernel count all of these so that it knows exactly how much memory has been committed. Then you run into two additional complications in practice: first, that much of this committed memory won't necessarily every get used (especially for socket buffers), and second that the kernel itself will need a certain amount of additional memory to process these things.
(The latter matters because the real goal with committed memory accounting is to make sure that the system never deadlocks or has to start killing things in order to make forward progress, and for that you need to include the kernel overhead. The former matters because people don't want to be restricted unless the system really is out of memory in some way.)
The good news is that many of these are just using memory; there are fewer that are committing memory that is not yet there (pipe buffers, for example, since POSIX specifies that you can write a certain amount of data to a pipe without blocking).