The ARC now seems to work right in ZFS on Linux
One of the long standing with ZFS on Linux has been its integration with the general Linux kernel memory allocation system. In particular, back in December I wrote about my problem of memory competition between the ZFS ARC and the general Linux page cache, where my much smaller ext4 root filesystem wound up with a lot of data cached and the ARC was much too small. The good news is that a lot can change in seven months.
As of some recent updates to the development version of ZFS on
Linux, this problem has vanished on my machine. I now consistently
see the ARC being a decent size and holding a significant amount
of data, and the ext4 page cache seems to be modest (you have to
sort of observe the ext4 page cache by implication, based on the
difference between ARC memory usage and what '
free' reports about
overall kernel buffer and cache usage). In short, the ZFS on Linux
ARC is now behaving like I expect a ZFS ARC to behave in general.
arcstat.py 1' also suggests that the ARC is being
pretty effective when I do things like compile Firefox from source.)
I don't know if all of ZFS on Linux's kernel memory issues are gone. I haven't run into any, but that isn't exactly conclusive (and I do look on my machine for signs of things like memory fragmentation). But I can say that ZoL is now handling the ARC much better than it used to and seems to be much more effective at stealing memory back from ext4. In general this strikes me as a good omen for ZoL really solving its long standing memory usage issues, which will make it a significantly safer thing to use.
Sidebar: Some numbers
Although my machine's current state isn't quite comparable to what
it was when I wrote my previous entry,
I'll give comparative numbers anyways. On the same 32 GB machine,
the ZFS ARC is currently using 16.6 GB with 12 GB of that being
file data. Assuming that all of the ZFS ARC size counts as general
kernel buffer/cache in '
free' output, the ext4 page cache is using
under a GB. This is quite different from the previous numbers
(although looking back I'm not entirely sure I was getting the size
of the ext4 page cache right).
(All ZFS memory in total, ARC and otherwise, seems to be around 17.75 GB right now. This is kernel memory allocated, not memory in use, although overall utilization is around 95% to 98%.)
(Writing this entry has made me painfully aware that I was and am
somewhat fuzzy on what
free's output really means and that I need to
/proc/meminfo fields themselves mean, since
both summarizing things and being selective.)