Confirming the behavior of file block sizes in ZFS
ZFS filesystems have a property called their
is usually described as something like the following (from here):
All files are stored either as a single block of varying sizes (up to the recordsize) or using multiple recordsize blocks.
A while back I wrote about using
zdb to peer into how ZFS stores
files on disk, where I looked into how ZFS
stored a 160 Kb file and specifically if it really did use two 128
Kb blocks to hold it, instead of a 128 Kb block and a 32 Kb block.
The answer was yes, with some additional discoveries about ZFS
compression and partial blocks.
Today I wound up wondering once again if that informal description of how ZFS behaves was really truly the case. Specifically, I wondered if there were situations where ZFS could wind up with a mixture of block sizes, say a 4 Kb block that was written initially at the start of the file and then a larger block written later after a big hole in the file. If ZFS really always stored sufficiently large files with only recordsize blocks, it would have to go back to rewrite the initial 4 Kb block, which seemed a bit odd to me given ZFS's usual reluctance to rewrite things.
So I did this experiment. We start out with a 4 Kb file, sync it,
zdb) that it's there on disk and looks like we expect,
and then extend the file with a giant hole, writing 32 Kb at 512
Kb into the file:
dd if=/dev/urandom of=testfile bs=4k count=1 sync [wait, check with zdb] dd if=/dev/urandom of=testfile bs=32k seek=19 count=1 conv=notrunc sync
The first write creates a
testfile that had a ZFS file block size
of 4 Kb (which
zdb prints as the
dblk field); this is the initial
conditions we expect. We can also see a single 4 Kb data block at
# zdb -vv -bbbb -O ssddata/homes cks/tmp/testfile [...] Indirect blocks: 0 L0 0:204ea46a00:1000 1000L/1000P F=1 B=5401327/5401327
After writing the additional 32 Kb,
zdb reports that the file's
block size has jumped up to 128 Kb, the standard ZFS dataset
recordsize; this again is what we expect. However, it also
reports a change in the indirect blocks. They are now:
Indirect blocks: 0 L1 0:200fdf4200:400 20000L/400P F=2 B=5401362/5401362 0 L0 0:200fdf2e00:1400 20000L/1400P F=1 B=5401362/5401362 80000 L0 0:200fdeaa00:8400 20000L/8400P F=1 B=5401362/5401362
The L0 indirect block that starts at file offset 0 has changed.
It's been rewritten from a 4 Kb logical / 4 Kb physical block to
being 128 Kb logical and 5 Kb physical (this is still an
pool), and the TXG it was created
B= field) is the same as the other blocks.
So what everyone says about the ZFS recordsize is completely true.
ZFS files only ever have one (logical) block size, which starts out
as small as it can be and then expands out as the file gets more
data (or, more technically, as the maximum offset of data in the
file increases). If you push it, ZFS will rewrite existing data
you're not touching in order to expand the (logical) block size out
to the dataset
If you think about it, this rewriting is not substantially different
from what happens if you write 4 Kb and then write another 4 Kb
after it. Just as here, ZFS will replace your initial 4 Kb data
block with an 8 Kb data block; it just feels more a bit more expected
because both the old and the new data falls within the first full
recordsize block of the file.
(Apparently, every so often something in ZFS feels sufficiently odd to me that I have to go confirm it for myself, just to be sure and so I can really believe in it without any lingering doubts.)