Here's something that's potentially very important, as it was for me today: if you're using an ordinary basic virtualization system (where you have guest OSes on top of a regular host OS), your virtual disks almost certainly have the performance characteristics of 4K physical sector size disks.

(In some situations they may have even bigger effective physical sector sizes.)

This happens because in a standard basic virtualization system, the guest OS disks are just files in a host OS filesystem and that host OS filesystem almost certainly has at least a 4 Kbyte basic block size. Sure the files are byte-addressable, but writing less than 4 Kb or writing things not aligned on 4Kb boundaries means that the host filesystem will generally have to do a read modify write cycle to actually write the guest's data out to its disk file (and then later do unaligned reads to get it back). Depending on how the virtualization system is implemented this can also require a whole bunch more memory copies as the VM hypervisor re-blocks and de-blocks data flowing between it and the host filesystem instead of just handing nice aligned 4 Kb pages off to the filesystem, where they will flow straight through to the hardware disk driver.

Under a lot of circumstances this won't actually matter. Many (guest) filesystems have a 4 Kb or bigger basic block size themselves and issue aligned IO in general, regardless of what they think the disk's (physical) block size is; if this is the case, the IO to the disk files in the host filesystem will generally wind up being in aligned 4 Kb blocks anyways. But if you have a guest OS filesystem that does not necessarily issue aligned writes, well, then this can make a real difference. ZFS is such a filesystem; if it thinks it's dealing with 512 byte sector disks it will issue all sorts of unaligned writes and reads.

The punchline to this is that today I doubled the streaming write and read IO speeds for my ZFS on Linux test VM with a simple configuration change (basically telling it that it was dealing with 4K disks). The IO speeds went from kind of uninspiring to almost equal to ext4 on the same (virtual) disk.

(My Illumos/OmniOS test VM also had its ZFS IO speed jump, although not as much; it was faster to start with for some reason.)