Virtual disks should be treated as 4k 'Advanced Format' drives

May 3, 2013

Here's something that's potentially very important, as it was for me today: if you're using an ordinary basic virtualization system (where you have guest OSes on top of a regular host OS), your virtual disks almost certainly have the performance characteristics of 4K physical sector size disks.

(In some situations they may have even bigger effective physical sector sizes.)

This happens because in a standard basic virtualization system, the guest OS disks are just files in a host OS filesystem and that host OS filesystem almost certainly has at least a 4 Kbyte basic block size. Sure the files are byte-addressable, but writing less than 4 Kb or writing things not aligned on 4Kb boundaries means that the host filesystem will generally have to do a read modify write cycle to actually write the guest's data out to its disk file (and then later do unaligned reads to get it back). Depending on how the virtualization system is implemented this can also require a whole bunch more memory copies as the VM hypervisor re-blocks and de-blocks data flowing between it and the host filesystem instead of just handing nice aligned 4 Kb pages off to the filesystem, where they will flow straight through to the hardware disk driver.

Under a lot of circumstances this won't actually matter. Many (guest) filesystems have a 4 Kb or bigger basic block size themselves and issue aligned IO in general, regardless of what they think the disk's (physical) block size is; if this is the case, the IO to the disk files in the host filesystem will generally wind up being in aligned 4 Kb blocks anyways. But if you have a guest OS filesystem that does not necessarily issue aligned writes, well, then this can make a real difference. ZFS is such a filesystem; if it thinks it's dealing with 512 byte sector disks it will issue all sorts of unaligned writes and reads.

The punchline to this is that today I doubled the streaming write and read IO speeds for my ZFS on Linux test VM with a simple configuration change (basically telling it that it was dealing with 4K disks). The IO speeds went from kind of uninspiring to almost equal to ext4 on the same (virtual) disk.

(My Illumos/OmniOS test VM also had its ZFS IO speed jump, although not as much; it was faster to start with for some reason.)

Comments on this page:

From at 2013-05-03 11:02:22:

That's why everyone backs VMs with LVM volumes, duh.

By cks at 2013-05-03 11:19:41:

It hadn't occurred to me to use LVM volumes for this, partly because the virtualization stuff I use makes files the default (and very easy to use; you basically don't need to do anything and they're thin provisioned by default). I suspect that a lot of casual virtualization is done this way.

(Heavy duty production VMs are likely to be done differently, with LVM or iSCSI or any number of other alternatives.)

From at 2013-05-04 03:45:56:

would you care to share the steps you took to do that? I mean, I assume you have a stock installation of omnios in kvm/virtualbox. So you cannot change the ashift setting when installing the OS (am I wrong there?). So how do you modify it afterwards?

By cks at 2013-05-05 01:03:38:

I'm afraid that my steps aren't going to be useful for your situation. The virtual disks I cared about were (virtual) iSCSI disks where I could set the advertised physical block size and I actually created my test pool with ZFS on Linux (which supports setting an explicit ashift, so I did that just to be sure). I don't think my particular VM system provides any way to actually advertise a 4K physical block size at the level of the guest OS talking directly to the disks (although clearly it should).

Sorry, I wish I had better news.

Written on 03 May 2013.
« Two xargs gotchas that you may not know about
What I see as RISC's big bets »

Page tools: View Source, View Normal, Add Comment.
Login: Password:
Atom Syndication: Recent Comments.

Last modified: Fri May 3 01:29:09 2013
This dinky wiki is brought to you by the Insane Hackers Guild, Python sub-branch.