How to do a very cautious LVM storage migration
A while back I wrote about how I was tempted by LVM mirroring when I wanted to migrate my LVM setup from a RAID mirror on some old disks to a new RAID mirror on some new(er) disks. Because I am some peculiar combination of cautious and daring, I gave in to this temptation recently. Now that the migration has more or less finished, it's time I reported in how it went and how to do this.
The short summary is using LVM mirroring to migrate my LVM volume group
from disk to disk worked without problems but the next time I need to do
this I will probably just use
pvmove, because establishing the actual
mirrors was achingly slow and the whole process was kind of a tedious
pain in the rear. I don't know if
pvmove would be faster, but I can
(The mirrors seemed to perform decently once they were synchronized. But initial synchronization of about 250 GB of data took literally days and it was not disk speed limited; LVM never drove the disks at full bandwidth or full IOPs/second rates.)
There are two advantages of using LVM mirroring instead of
I used both of them. First, you can run for a while on both the new
storage and the old storage at the same time, to build up confidence in
the new storage. Second, you can preserve a complete and usable copy of
all of your data on the old storage, a copy that you can inspect, mount,
and so on if you wind up having to. With
pvmove, your data just moves;
you wind up only on the new storage and there's nothing left on the old
I read a number of writeups of how to do LVM mirroring on the web, but I
found all of them to be a little bit unclear (partly because the logic
of when you specified which disk device wasn't always clear). So here
is the annotated steps that I used. First, let's say that the old disk
space you're migrating away from is
/dev/OLD and the new disk space
/dev/NEW, and you're migrating the LVM volume group
vg0 with the
vg0/data, mounted on
/dev/NEWas a LVM physical volume:
- Add it to the volume group:
vgextend vg0 /dev/NEW
- Mirror each volume/filesystem to the new storage:
lvconvert -m1 --mirrorlog mirrored --alloc anywhere vg0/data /dev/NEW
This is the step that takes forever, and you have to repeat it for each filesystem (I did not try to
lvconvertmultiple volumes at once, I did them one at a time).
It's possible that you will not need '
--alloc anywhere'; leave it out the first time to see (if you do need it, LVM will report that it can't find space to put stuff). The important arguments are
-m1, which tells lvconvert to create a mirror (on
/dev/NEW, because that's the physical volume we specified) and
--mirrorlog mirroredwhich tells it to create a (mirrored) persistent on-disk log of what bits of the mirror are in sync.
If I was doing this again I might just use
--mirrorlog disk, because as it happens LVM put both of my mirror log mirrors on
/dev/NEWfor its own inscrutable reasons (it's possible that
--alloc anywhereinfluenced this). I didn't let this worry me because the whole situation was temporary and
/dev/NEWwas itself a mirrored RAID array, so it was already pretty reliable.
(It's possible that a non-mirrored mirrorlog would speed things up.)
- Verify that everything looks good:
lvs -a -o+devices
What this should show is that
vg0/datanow has four internal subvolumes. The
_mimage_Nsubvolumes are the actual mirrors (the original volume you started with and the mirror on the new storage), one on each of
/dev/NEW, and you'll also have two additional subvolumes for the mirror log (ideally one on each disk, but see above).
At this point you can run with full mirroring for as long as you want in order to build up confidence in the new disk(s). Once you're fully happy with them, it's time to complete the migration by splitting off the old disks.
- Split apart each volume, leaving the live version on the new disk and
creating a new volume that is the data on the old disk. I think
that I read that this apparently goes better if the filesystem
is unmounted at the time, so that's how I did it:
lvconvert --splitmirrors 1 -n data-o vg0/data /dev/OLD
-n data-ogives the volume name of the 'new' volume (ie, the name you want for the original volume on the original disk). We specify
/dev/OLDhere to tell lvconvert that it should act on the mirror side that is on
If you run '
lvs -a -o+devices' afterwards, you should see that all of those internal subvolumes have disappeared and you now have two volumes; vg0/data should be entirely on /dev/NEW and vg0/data-o should be entirely on /dev/OLD.
- After doing this for each filesystem you have one volume group using
both /dev/OLD and /dev/NEW but all of your live volumes are on
/dev/NEW; all of the volumes on /dev/OLD are unused. The final step is
to split apart the volume group itself into two, the live one on /dev/NEW
and a second volume group that is just all of the old volumes on /dev/OLD.
First, we need to make all of the volumes on /dev/OLD inactive:
lvchange -an vg0/data-o
This should complete without complaints because none of these volumes should be in use; they should all be quiescent, unmounted, and so on.
Then we can split the volume group itself:
vgsplit vg0 vg0-o /dev/OLD
vg0-ois the name of the 'new' volume group, ie the old copy of the data on the old storage. We specify
vgsplitto act on the volumes (and physical volume and so on) on
lvs -a -o+devices' should now show two volume groups, with vg0 using only
/dev/NEWand vg0-o using only
After this is done you can decommission
vg0-o at your leisure. I
haven't gotten around to doing that since I haven't quite reached the
point where I want to physically remove the old disks (I still have
my boot partition on them, partly because I need to figure out which
physical SATA plug on the motherboard actually is
sdb, and so
(I don't know if you can just disconnect the disks without doing
anything special in LVM. That would be the ideal way to do it since it
vg0-o and its volumes completely intact for any future
need, but LVM might get upset when you reboot your machine because a
volume group it expects isn't there.)