Wandering Thoughts archives

Moving a libvirt-based virtualization setup from one machine to another

Following my successful experience with libvirt on my desktop, I decoded to build out a general virtualization server for tests, scratch machines, and so on. Unfortunately, the old server I reused for this turned out not to be up to the task, at least not if I wanted it to be attractive. So today I reinstalled things on a newer, better server and wanted to move over the entire libvirt setup from the initial machine so that I didn't have to redo everything. What I needed to copy (and why) for this wasn't obvious to me, so here are my notes.

(My initial VM host was one of our old OmniOS ZFS NFS fileservers. I feel oddly sad that we're not using those machines for anything any more; they were decently nice machines in their time. Although since their KVM over IP console is Java-only and doesn't work with Fedora's normal Java, they've gotten less attractive.)

First off, you need to copy over the actual disk images (and ISO images). Libvirt would like to put these in /var/lib/libvirt/images, but I had relocated these to a new filesystem (which I called /virt) on different disks. So I rsync'd the entire thing over. Right now this isn't much data, since I haven't done much with virtual machines on this VM host (partly because, well, it's slow).

I think that the fast way to do the rest would have been to copy over /etc/libvirt and /var/lib/libvirt in their entirety. This would have left me needing to edit the libvirt configuration for my bridged network, because the network device names changed between the old hardware and the new hardware. Otherwise, I optimistically think I might not have had to touch anything else. However, I opted to do things piece by piece.

What I did and copied was more or less:

1. I used virt-manager to stop the default storage pool and (NAT) networking, and rename both of them.
2. I dumped the XML for the storage pools on the old server with 'virsh pool-dumpxml' and then created the pools with this XML and 'virsh pool-define'.
3. I'd kept the original XML definitions of my bridged networking on the old machine, so I edited it to have the right network device names and loaded the XML to define things with 'virsh net-define'.

4. The basic XML definitions for virtual machines are in /etc/libvirt/qemu with obvious names, so I copied them over. After rebooting (to restart the libvirt daemons from the ground up), this got me the VMs but not their snapshots.

5. To get the snapshots as well, I had to copy over /var/lib/libvirt/qemu/snapshot, which contains the XML defining the snapshots (I make all of my snapshots with the VM powered down).

The actual disk snapshots are already present as 'internal snapshots' in the VM QEMU disk images, so the XML just tells libvirt about them and lets you manipulate snapshots through libvirt tools like virsh instead of having to use raw qemu-img commands (the way you have to do with UEFI based VMs).

The libvirt storage is defined in XML files in /etc/libvirt/storage. Networking (for QEMU based VMs) is in /etc/libvirt/qemu/networks. You may have libvirt hook scripts in /etc/libvirt/hooks (as I do on my desktop to set up NAT). There's some dnsmasq stuff in /var/lib/libvirt/dnsmasq but I think it's probably all automatically written, or at least stuff that you can probably not care about copying over.

I don't regret going through the piece by piece effort because I feel that I now understand libvirt a bit more and I'm better equipped to poke around it for other things. But if I was doing it again I'd probably just copy all of /etc/libvirt and /var/lib/libvirt, then edit any network device names and so on as necessary.