2007-07-20
The downside of a unified buffer cache
When vendors started releasing Unix systems with a unified buffer cache, people discovered that the new operating system releases tended to perform not so well under load. Generally what happened is that programs got paged out madly, so any time you typed a new command line in your shell session there was a great big lurch as your shell was paged back in and then the program started and so on, and by the time you got back to your shell it had paged out again.
The underlying problem is that processes can use up pages through
read() a lot faster than they can use up pages through virtual memory,
because read() can touch a whole lot of pages in a single action while
actual executing code only touches a page at a time. The net result was
that programs doing disk IO basically ate the rest of the system; the
effect that was particularly bad on multi-user systems, where something
is doing disk IO pretty much all the time.
(Early versions of SunOS 4 exacerbated the situation by defaulting to maintaining only a remarkably small amount of free memory, so that merely starting a new command was virtually guaranteed to spark a page storm.)
Naturally, this was a hot topic for systems research in the early 1990s and produced a number of papers pointing out the issue, which was at least reassuring in that people understood why our (then) new systems were performing so badly, even if they didn't have a good solution.
(The situation has not gotten much better since then; virtual memory tuning problems still bedevil modern Unixes like FreeBSD and Linux. Many of the issues even look eerily familiar.)
2007-07-02
What the unified buffer cache is unifying
Pretty much all Unixes these days have what is called a unified buffer cache. If you don't know something about Unix history, this name is a little puzzling, because what is it unifying?
The simple explanation is that originally Unix had the buffer cache,
which cached blocks of recent disk IO (whether directly from user
processes via read() and write(), or from internal kernel IO), and
process virtual memory, used for the code and data of running processes.
The buffer cache was statically sized, generally at 10% of the physical
RAM in the machine; 'ordinary RAM', used for processes, was whatever
you had left after the kernel was done allocating its data structures
(including the buffer cache).
(This split made sense in a world without mmap() because there was a
clear separation between 'process virtual memory', only directly used by
processes, and 'disk IO buffers', which were only directly used by the
kernel.)
The static sizing of the buffer cache didn't please people any more than simple swap space assignment did, since it was quite possible to have significant amounts of your system's RAM be inactive and thus more or less wasted. Researchers and Unix vendors promptly got to work on unifying the buffer cache and process virtual memory, so that they both shared the same pool of RAM and each could potentially use (nearly) all of it.
(I believe the first vendor to deliver a system with a unified buffer
cache was Sun with SunOS 4, which also introduced mmap(). You could
say that mmap() forced Sun's hand on this, since systems with mmap()
more or less erase the clear distinction between process virtual memory
and kernel level disk IO buffers by having some pages that are both.)