Trying to understand the ZFS
If you read the Illumos ZFS source code or perhaps some online guides
to ZFS performance tuning, you may run across mention of a tunable
l2arc_noprefetch. There are various explanations of what
this tunable means; for example, the current Illumos source code
boolean_t l2arc_noprefetch = B_TRUE; /* don't cache prefetch bufs */
As you can see, this defaults to being turned on in Illumos (and in ZFS on Linux). You can find various tuning guides online that suggest turning it to off for better L2ARC performance, and when I had an L2ARC in my Linux system I ran this way for a while. One tuning guide I found describes it this way:
This tunable determines whether streaming data is cached or not. The default is not to cache streaming data. [...]
This makes things sound like you should absolutely turn this on, but not so fast. The ZFS on Linux manpage on these things describes it this way:
Do not write buffers to L2ARC if they were prefetched but not used by applications.
That sounds a lot more reasonable, and especially it sounds reasonable to have it turned on by default. ZFS prefetching can still be overly aggressive, and (I believe) it still doesn't slow itself down if the prefetched data is never actually read. If you are having prefetch misses, under normal circumstances you probably don't want those misses taking up L2ARC space; you'd rather have L2ARC space go to things that you actually did read.
As far as I can decode the current Illumos code, this description
also seems to match the actual behavior. If a ARC header is flagged
as a prefetch, it is marked as not eligible for the L2ARC; however,
if a normal read is found in the ARC and the read is eligible for
L2ARC, the found ARC header is then marked as eligible (in
arc_read()). So if you prefetch then get a read hit, the ARC
buffer is initially ineligible but becomes eligible.
If I'm reading the code correctly,
l2arc_noprefetch also has a
second, somewhat subtle effect on reads. If the L2ARC contains the
block but ZFS is performing a prefetch, then the prefetch will not
read the block from the L2ARC but will instead fall through to doing
a real read. I'm not sure why this is done; it may be that it
simplifies other parts of the code, or it may be a deliberate attempt
to preserve L2ARC bandwidth for uses that are considered more likely
to be productive. If you set
l2arc_noprefetch to off, prefetches
will read from the L2ARC and count as L2ARC hits, even if they are
not actually used for a real read.
Note that this second subtle effect makes it hard to evaluate the
true effects of turning off
l2arc_noprefetch. I think you can't
go from L2ARC hits alone because L2ARC hits could be inflated by
prefetching, putting the prefetched data in L2ARC, throwing it away
before it's used for a real read, re-prefetching the same data and
getting it from L2ARC, then throwing it away again, still unused.