== What _free_ is really telling you, and some bits from _/proc/meminfo_

The _free_ command is a pretty venerable Linux sysadmin tool; often
it's one of the first things I reach for when I suspect that a
system might be having RAM problems. Its normal output certainly
looks straightforward enough, but [[as I discovered recently
ZFSOnLinuxARCNowWorksRight]] it's always good to actually look up
the details of this stuff.

So let's start with the normal output:

.pn prewrap on
 ; free -m
       total   used   free  shared  buff/cache  available
 Mem:  32158  13111    683      33       18363       9727
 Swap:  4094      0   4094

Total is the total physical memory. 'Used' is a calculated figure;
according to the manpage, it's _total_ minus _free_ and _buff/cache_
(and does not include _shared_). _buff/cache_ is the sum of what
'_free -w_' will show separately as _buffers_ and _cache_, and this
is where life gets interesting.

Free's _buffers_ is _/proc/meminfo_'s _Buffers_ field,
which according to [[the fine documentation
https://www.kernel.org/doc/Documentation/filesystems/proc.txt]] is:

> Relatively temporary storage for raw disk blocks \\
> shouldn't get tremendously large (20MB or so)

That size info is clearly what they call 'a little bit out of
date', considering that our normal machines run around 400 MB of
_Buffers_. There can be variations from this, and in particular my
machine running ZFS on Linux usually has substantially more space in
_Buffers_; right now it's almost 4 GB.

Free's _cache_ is the sum of meminfo's _Cached_ and _Slab_ fields.
_Cached_ is the page cache, which is used by normal filesystems to
hold pages read off disk (or in the case of NFS, off the network).
_Slab_ reports the amount of memory used by the kernel's [[slab
allocator https://en.wikipedia.org/wiki/Slab_allocation]]. Calling
this _cache_ is potentially misleading, since the slab allocator
is widely used all through the kernel for many, many things as you
can see by looking at _/proc/slabinfo_ or running _slabtop_, in
both cases as root (note that [[slab merging SlabinfoSlabMerging]]
can make the slab names somewhat misleading). However it's true
that some of the kernel's slab memory usage is for 'caches' (broadly
construed).

Free's _available_ is meminfo's _MemAvailable_, which is a kernel
estimate based on a bunch of internal heuristics that I'm not going
to try to write up. These heuristics can be wrong for various
reasons, including kernel subsystems that can shrink their slab
usage under memory pressure but don't put the right magic markers
on their slabs so that some of their slab usage shows up in
_MemAvailable_.

Nothing in _free_'s output tells you how much memory the kernel is
using in total. The kernel uses more memory than appears in _buffers_
and _cache_, and _free_ does not attempt to reverse engineer a
kernel usage figure based on finding out how much memory appears
to be in use by user programs. This is fair enough, given that
_free_ is primarily concerned with telling you how much memory is
or might be freeable (it's in its name, after all). Still, it's
something to bear in mind; if you really want to get a picture of
kernel memory usage, you want another tool.

(I don't know what tool you want; I haven't looked yet. I suspect
that sufficient information is exposed in _/proc/meminfo_ to work
this out with reasonable accuracy.)

PS: If it was up to me, I would redefine free's _cached_ to be
meminfo's _Cached_ plus _SReclaimable_. The latter is the amount
of space used by slabs that have explicitly marked themselves as
reclaimable under memory pressure, which is hopefully more or less
the set of slabs that are actually being used as caches instead of
for more sticky things.