2007-07-20
The downside of a unified buffer cache
When vendors started releasing Unix systems with a unified buffer cache, people discovered that the new operating system releases tended to perform not so well under load. Generally what happened is that programs got paged out madly, so any time you typed a new command line in your shell session there was a great big lurch as your shell was paged back in and then the program started and so on, and by the time you got back to your shell it had paged out again.
The underlying problem is that processes can use up pages through
read() a lot faster than they can use up pages through virtual memory,
because read() can touch a whole lot of pages in a single action while
actual executing code only touches a page at a time. The net result was
that programs doing disk IO basically ate the rest of the system; the
effect that was particularly bad on multi-user systems, where something
is doing disk IO pretty much all the time.
(Early versions of SunOS 4 exacerbated the situation by defaulting to maintaining only a remarkably small amount of free memory, so that merely starting a new command was virtually guaranteed to spark a page storm.)
Naturally, this was a hot topic for systems research in the early 1990s and produced a number of papers pointing out the issue, which was at least reassuring in that people understood why our (then) new systems were performing so badly, even if they didn't have a good solution.
(The situation has not gotten much better since then; virtual memory tuning problems still bedevil modern Unixes like FreeBSD and Linux. Many of the issues even look eerily familiar.)