Trying to understand the ZFS l2arc_noprefetch tunable

July 24, 2017

If you read the Illumos ZFS source code or perhaps some online guides to ZFS performance tuning, you may run across mention of a tunable called l2arc_noprefetch. There are various explanations of what this tunable means; for example, the current Illumos source code for arc.c says:

boolean_t l2arc_noprefetch = B_TRUE; /* don't cache prefetch bufs */

As you can see, this defaults to being turned on in Illumos (and in ZFS on Linux). You can find various tuning guides online that suggest turning it to off for better L2ARC performance, and when I had an L2ARC in my Linux system I ran this way for a while. One tuning guide I found describes it this way:

This tunable determines whether streaming data is cached or not. The default is not to cache streaming data. [...]

This makes things sound like you should absolutely turn this on, but not so fast. The ZFS on Linux manpage on these things describes it this way:

Do not write buffers to L2ARC if they were prefetched but not used by applications.

That sounds a lot more reasonable, and especially it sounds reasonable to have it turned on by default. ZFS prefetching can still be overly aggressive, and (I believe) it still doesn't slow itself down if the prefetched data is never actually read. If you are having prefetch misses, under normal circumstances you probably don't want those misses taking up L2ARC space; you'd rather have L2ARC space go to things that you actually did read.

As far as I can decode the current Illumos code, this description also seems to match the actual behavior. If a ARC header is flagged as a prefetch, it is marked as not eligible for the L2ARC; however, if a normal read is found in the ARC and the read is eligible for L2ARC, the found ARC header is then marked as eligible (in arc_read()). So if you prefetch then get a read hit, the ARC buffer is initially ineligible but becomes eligible.

If I'm reading the code correctly, l2arc_noprefetch also has a second, somewhat subtle effect on reads. If the L2ARC contains the block but ZFS is performing a prefetch, then the prefetch will not read the block from the L2ARC but will instead fall through to doing a real read. I'm not sure why this is done; it may be that it simplifies other parts of the code, or it may be a deliberate attempt to preserve L2ARC bandwidth for uses that are considered more likely to be productive. If you set l2arc_noprefetch to off, prefetches will read from the L2ARC and count as L2ARC hits, even if they are not actually used for a real read.

Note that this second subtle effect makes it hard to evaluate the true effects of turning off l2arc_noprefetch. I think you can't go from L2ARC hits alone because L2ARC hits could be inflated by prefetching, putting the prefetched data in L2ARC, throwing it away before it's used for a real read, re-prefetching the same data and getting it from L2ARC, then throwing it away again, still unused.

Comments on this page:

By Indivar Nair at 2019-10-04 06:42:33:

Hi Chris,

Nice article.

I have a couple of queries -

1. Though it wasn't intended to do so, but does setting "l2arc_noprefetch" to OFF ultimately help cache sequential files?

2. You mentioned that - "If the L2ARC contains the block but ZFS is performing a prefetch, then the prefetch will not read the block from the L2ARC but will instead fall through to doing a real read"

Is this applicable even if the whole file is in L2ARC? Will ZFS still fetch it from the disk and not L2ARC?

By cks at 2019-10-04 14:10:40:

With the disclaimer that it's been a few years since I looked at this, I believe the answers to your questions are yes (theoretically) and no.

Setting l2arc_noprefetch off pushes things into L2ARC more aggressively, so it can potentially help cache files that will be read sequentially sooner or later. However, if you always fully read files that you open (whether sequentially or randomly), it won't make a difference. Any data that you read at the user level is always eligible for going into the L2ARC, whether or not it was initially prefetched.

I just took a quick look at the current code, and as far as I can tell all prefetch reads skip the L2ARC even if the data is present in it (assuming that l2arc_noprefetch is set to on, the default). One of the conditions for reading from the L2ARC is:

!(l2arc_noprefetch && HDR_PREFETCH(hdr))

The inner condition is true if this is a prefetch and l2arc_noprefetch is on, which means that the negated version is false in that case and it won't read from the L2ARC. (This is in arc.c deep in the depths of arc_read. There's an associated code comment, but as far as I can tell it's slightly wrong about the conditions here.)

Written on 24 July 2017.
« Understanding a bit about the SSH connection protocol
If you're going to use PyPy, I think you need servers »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Mon Jul 24 02:06:57 2017
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.