Setting the ARC target size in ZFS on Linux (as of ZoL 2.1)

April 20, 2023

In the past I've grumbled about wanting a way to explicitly set the (target) ARC size. After all of my recent investigation into how the ARC grows and shrinks, I now believe that this can be safely done, at least some of the time. However, growing (or in general resizing) the ZFS ARC comes with a number of caveats, because it's only going to be effective some of the time.

The simple and brute force way to grow the ARC target size to a given number is to briefly and temporarily raise zfs_arc_min to your desired value, which can be done through /sys/module/zfs/parameters. After having spent some time going through the ARC code, I'm relatively convinced that this is safe and won't trigger immediate consequences. You can similarly reduce the ARC target size by (temporarily) lowering zfs_arc_max. In both cases this has an immediate effect on 'c', the ARC target size; when you set either the maximum or the minimum, the ZFS code immediately sets 'c' if it's necessary.

However, raising the ARC target size will only have a meaningful effect if ZFS can actually use more memory. If the free memory situation is bad enough that memory_available_bytes is negative, your newly set ARC target size will pretty much immediately start shrinking, possibly significantly, and the ARC will have no chance to actually use much more extra memory. If available memory is positive but not very large, it may turn negative once the ARC's actual size grows a bit more and then ZFS will shrink your recently-raised ARC target size back down, along with probably shrinking the ARC's actual memory use.

Given all of this, there seem to be two good cases to deliberately raise the ARC target size. The first case is if you've seen an odd collapse in the ARC target size and you have a lot of free memory. Here the ARC target size will probably grow on its own, eventually, but it will likely do that in relatively small increments (such as 128 KiB at a time), while you can yank it right up now. The second case is if the ARC target size is already quite big but arc_no_grow is stuck at '1' because ZFS wants an extra 1/32nd of your large target size to be available; this is probably more likely to be an issue if you've raised zfs_arc_max (as we have on our fileservers).

(As far as I can tell from looking at the code, arc_no_grow being 1 doesn't prevent the ARC from allocating extra memory to grow up to the ARC target size; it just prevents the ARC target size from growing further.)

In theory you can lock the ARC target size at a specific value by boxing it in by setting zfs_arc_min to sufficiently close to zfs_arc_max. While this will keep ZFS from lowering the target size, it won't keep either ZFS or the general kernel 'shrinker' memory management feature from frantically trying to reclaim memory from the ARC if actual available memory isn't big enough. Fighting the kernel is probably not going to give you great results.

Written on 20 April 2023.
« An interesting mistake I made with a (Go) SSH client API
The traditional workaround for stuck NFS(v3) locks »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Thu Apr 20 22:58:04 2023
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.