== Limiting a process's memory usage on Linux

Due to [[recent events 64BitDrawback]] I have become interested in
this issue, so I have been poking around and doing some experiments.
Unfortunately, while Linux has a bewildering variety of memory related
per-process resource limits that you can set, most of them don't work or
don't do you any good.

What you have, in theory and practice:

* _ulimit -m_, the maximum RSS, doesn't do anything; the kernel
  maintains the number but never seems to use it for anything.

* _ulimit -d_, the maximum data segment size, is effectively useless
  since it only affects memory that the program obtains through
  _brk(2)/sbrk(2)_. These days, these aren't used very much; GNU libc
  does most of its memory allocation using _mmap()_, especially for
  big blocks of memory.

* _ulimit -v_, the maximum size of the address space, works but
  affects *all* _mmap()_s, even of things that will never require
  swap space, such as _mmap()_ing a big file.

What I really want is something that can effectively limit a process's
'committed address space' (to use the term that _/proc/meminfo_ and the
kernel documentation on swap overcommit use). I don't care if a process
wants to _mmap()_ a 50 gigabyte file, but I care a lot if it wants 50G
of anonymous, unbacked address space, because the latter is what will
drive the system into out-of-memory.

Unfortunately I can imagine entirely legitimate reasons to want to
_mmap()_ huge files (especially huge sparse files) on a 64-bit machine,
so any limit on the total process address space on our compute servers
will have to be a soft limit.

Since the Linux kernel already tracks committed address space
information for the whole system, it's possible that it would not be too
much work to extend it to a per-process limit. (The likely fly in the
ointment is that memory regions can be shared between processes, which
complicates the accounting and raises questions about what you do when a
process modifies a virtual memory region in a way that is legal for it
but pushes another process sharing the VMA over its limit.)