2012-01-15
Understanding the basic shape of Unix virtual memory management
Although their implementations vary in detail, every modern Unix system
(ie, everyone with mmap()) has some basic constraints that shape the
general outline of their virtual memory system. While I'm talking about
virtual memory statistics, it's worth running down
this basic shape and what creates it.
(I say this partly because I spend part of writing yesterday's entry sorting bits of this out in my own head.)
The fundamental mmap() operation is to glue some resource (such as a
portion of a file) into the process's memory address space. Multiple
processes can each map the same resource into their memory space; when
they do, all processes need to share the same pages of physical RAM in
order to keep everyone's view of the resource in sync (in fact you want
this to happen for write() based IO as well). This implies that a
process's address space is effectively composited together from a bunch
of entities representing different memory areas.
However, this compositing needs to involve a layer of indirection. Processes don't all map a resource at the same address space (for example, a shared library may be mapped at many different addresses), plus processes can overlay private changes on top of shared resources (eg, copy on write for various things). This implies a two step mapping; a process has 'virtual memory areas' holding information used to build its own page table (and to track private versions of pages), and these then point to a shared data structure to keep track of the actual resource, what pages it has in RAM, and so on.
(If we construe 'process' broadly, processes sometimes share VMAs; threads traditionally run in a shared address space, for example.)
In theory, Unix systems could have what I will call coherent page table mappings of resources, where if a single process brought a page of a shared resource into RAM all processes using that resource would get a page table entry for that page. In practice, this would involve a lot of page table changes for processes that don't care about the pages in question and may never refer to it, so I think that basically all Unix processes have incoherent mappings; a page of a resource may be in RAM and be mapped by other processes using it without this process having a PTE for it. When your process tries to refer to that page it will take a soft page fault to establish the PTE it needs and then immediately go on.
(The traditional distinction between soft page faults and hard page faults is that a hard page fault needs to fetch things from disk while a soft page fault just updates things in memory.)
In turn this means that a page of a shared resource can be taken away from your process in one of two ways; call these 'local' and 'global'. A 'local' removal just removes your PTE for the page, but it stays mapped by any other processes that are using it. A 'global' removal makes the page unreferenced by anyone by removing all PTEs in all processes that were (still) using it. Normally you hope to get rid of pages by a series of local removals that result in the page not being mapped by anyone, at which point you can do a free global removal.
This obviously complicates efforts to answer questions about memory usage where it involves shared resources. A shared resource will be using some number of pages of RAM, but not all of those pages will be mapped in any particular process or group of processes. In fact it may not be possible to determine how many active pages a particular resource has because your system may only give you a 'per-process' view of shared resources.
2012-01-14
What do we mean when we talk about something's memory usage?
It should be possible for a reasonably intelligent human being (in which category I place myself) to answer simple questions about system memory usage, such as "How much memory is my database using?" or "How much memory is my web server using?" relatively simply.
One of the problems is that the question being asked here is not well defined. There are several things that this question could mean (talking only about a single process in order to simplify life):
- how much virtual memory the process has asked the operating system for.
- how much virtual memory the process has actually looked at; processes
often ask for more memory than they ever use.
(Often this isn't deliberate; for example, a low level memory allocator may ask for extra space because it anticipates future requests that turn out to never get made.)
- how much RAM would be freed up if this process didn't exist, which is theoretically the same as how much extra RAM this process uses as it is running.
- how much RAM the process would require if it was the only thing
running on the system (or at least if it shared nothing with any
other process on the system).
- what the process's 'fair share' of all of the RAM in use on the system
is, where some attempt is made to assign a portion of the cost
of RAM that's being shared between several processes to each
process.
(As hinted by the two previous questions, this 'fair share' idea is somewhat artificial; it's quite possible that all of the shared RAM would still be used and needed even if this process didn't exist.)
- how much memory would be required if the operating system had to make
good on all of its various promises of memory to the process (including
for all of the copy-on-write memory that the process could theoretically
write to), what we can call committed address space.
(This is generally unrealistically pessimistic, but not always.)
(Each of these questions is useful and interesting in certain situations.)
What makes most of these questions difficult and complicated is memory that's shared between processes. If there was no memory sharing (or only negligible memory sharing) then several of the questions would collapse together and it would be easy for the operating system to give useful answers to most of them. Unfortunately for Robert Haas, modern Unix systems and modern applications share significant amounts of memory in many circumstances.
(To be fair, properly accounting for shared memory usage has bedeviled
Unix from the moment people implemented copy on write for fork().)
There are theoretically straightforward extensions of all of these questions to groups of processes. For things like 'how much RAM would be freed up if they all exited', you have to work out what RAM or virtual memory is shared only between all of the processes versus what RAM is also (partially) shared with outside processes. RAM used only within the group gets entirely charged to the group; RAM also shared outside the group may need to be handled in various ways depending on the specific question you're asking.
(Correctly and usefully grouping processes together is also often not a completely trivial issue. What processes should be considered to be 'your web server' or 'your database server' is often something that's obvious to an experienced human but not necessarily something that's clear to a computer in any useful way. Even when you can come up with an acceptable mechanical definition of a group, groups can easily overlap or be supersets of each other; consider the groups of 'all processes executing this binary' and 'all processes descending from pid <X>'.)
By the way: you may need very low-level access to page table information in order to get correct answers to these questions for groups of processes. If the system provides information on who has what memory areas mapped it's relatively easy to detect entire memory areas that are only shared within a group (eg, all of the database server processes are the only users of a common shared memory segment). But to detect the case where some area of a broadly-shared object is only used by your group of processes, you need detailed per-page information.
(For instance, your database server processes might be the only users of a set of functions and data tables in the base C++ support library, although lots of other processes also have the library mapped.)