ZFS DVAs and what they cover on raidz vdevs

August 21, 2022

In ZFS, a DVA (Device Virtual Address) is the equivalent of a block address in a regular filesystem. For our purposes today, the important thing is that a DVA tells you where to find data by a combination of the vdev (as a numeric index) and an offset into the vdev (and also a size). This implies, for example, that in mirrored vdevs, all mirrors of a block are at the same place on each disk, and that in raidz vdevs the offset is striped sequentially across all of your disks.

Recently I got confused about one bit of how DVA offsets work on raidz vdevs. On a raidz vdev, is the offset an offset into the logical data on the vdev (which is to say, ignoring and skipping over the space used by parity), or is it an offset into the physical data on the vdev (including parity space)?

The answer is on raidz vdevs, DVA offsets cover the entire available disk space and include parity space. If you have a raidz vdev of five 1 GB disks, the vdev DVA offsets go from 0 to 5 GB (and by corollary, the parity level has no effect on the meaning of the offset). This is what is said by, for example, Max Bruning's RAIDZ On-Disk Format.

If you think about how parity works in raidz vdevs, this makes sense. In raidz, unlike conventional RAID5/6/7, the amount of parity and its location isn't set in advance, and you can force ZFS to make 'short writes' (writes of less than the data width of your raidz vdev) that still force it to create full parity. If DVA offsets ignored and skipped over parity, going from an offset to an on disk location would be very complex and the maximum offset limit for a vdev could change. Having the DVA offset cover all disk space on the vdev regardless of what it's used for is much simpler (and it also allows you to address parity with ordinary ZFS IO mechanisms that take DVAs).

I don't know how DVA offsets work on draid vdevs. Since I haven't been successful at understanding the draid code's addressing in the past, I haven't tried delving into it this time around.

(I might have known this at some point when I first looked into ZFS DVAs, but if so I forgot it since then, and recently had a clever ZFS idea that turns out to not work because of this. Now that I've written this down, maybe I'll remember it for the next time around.)

Written on 21 August 2022.
« The Ubuntu 22.04 server installer wants you to scrub reused disks first
Some notes on Grafana annotations sourced from Prometheus metrics »

Page tools: View Source, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Sun Aug 21 21:52:22 2022
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.