Migrate a VM with Minimal Downtime

• Migrate a VM with Minimal Downtime

This script takes a snaphot of the running VM, which leaves the qcow2 backing images in a known state. All VM activity is then recorded in the snapshots, while the backing images are rsync'd to a reciprocal hypervisor on the 1G VLAN.

After the backing images are transferred, the VM is shutdown and snapshots copied to the reciprocal hypervisor. The server is defined on the new host, started, and the snapshots are merged into the backing image/s.

Total downtime depends on the size of the snapshots, but is usually less than a minute or two. The whole process takes about an hour an a half for a ~200GB QCOW2 image over a 1GB/sec VLAN.

  Example 1:

jeff@sheri:/var/lib/libvirt/images/scripts$ cat skillsregistry.migrate.sh 
#!/bin/bash
################################################################################
######################  Migrate VM to a new hypervisor  ########################
# Authors: Jeff Brown / Lucio de Re
# Date: 2019/11/26
#
# This script takes a snaphot of the running VM, which leaves the qcow2 backing
# images in a known state. All VM activity is then recorded in the snapshots,
# while the backing images are rsync'd to a reciprocal hypervisor on the VLAN.
# After the backing images are transferred, the VM is shutdown and snapshots
# copied to the reciprocal hypervisor. The server is defined on the new host,
# started, and the snapshots are  merged into the backing image/s.
# Total downtime depends on the size of the snapshots, but is usually
# less than a minute or two. The whole process takes about an hour and a half.
#
# NB. This script must be run by a user who is a member of libvirt-qemu group.
# NB. The user must also be able to run sudo aa-teardown/chown/chmod without password.
# E.g. jeff ALL=NOPASSWD:/bin/chown, /bin/chmod
#
# 192.100.0.4 (vespa.org.za)
# 192.100.0.82 (abispa.org.za)
#
################################################################################

cd /var/lib/libvirt/images/
virsh snapshot-create-as --domain skillsregistry skillsregistry.snapshot --diskspec sda,file=/var/lib/libvirt/images/snapshots/skillsregistry.snapshot.qcow2 --disk-only --atomic
export RSYNC_PASSWORD="real password for rsyncd secret"
rsync -ShavW --progress /var/lib/libvirt/images/skillsregistry.qcow2 rsync://tergum@192.168.100.82/bvar/lib/libvirt/images/skillsregistry.qcow2
virsh shutdown skillsregistry

x=0
while sleep 5; do
virsh list --all | grep skillsregistry | grep "shut off" && break
x=$(expr $x + 1)
[ $x -gt 120 ] && { echo >&2 timed out after $(expr $x \* 5) seconds; exit 3; }
done

virsh autostart skillsregistry --disable
virsh -c qemu:///system dumpxml skillsregistry > /var/lib/libvirt/images/xml/skillsregistry.xml
scp /var/lib/libvirt/images/xml/skillsregistry.xml 192.168.100.82:/var/lib/libvirt/images/xml/
sudo find . -type f -exec chown libvirt-qemu:libvirt-qemu {} \; -exec chmod g+rw {} \;
rsync -ShavW --progress /var/lib/libvirt/images/snapshots/skillsregistry.snapshot.qcow2 rsync://tergum@192.168.100.82/bvar/lib/libvirt/images/snapshots/

export RSYNC_PASSWORD="nopass"

virsh -c qemu+ssh://192.168.100.82/system define /var/lib/libvirt/images/xml/skillsregistry.xml
virsh -c qemu+ssh://192.168.100.82/system start skillsregistry

x=0
while sleep 5; do
virsh -c qemu+ssh://192.168.100.82/system list --all | grep skillsregistry | grep "running" && break
x=$(expr $x + 1)
[ $x -gt 120 ] && { echo >&2 timed out after $(expr $x \* 5) seconds; exit 3; }
done

# Required for Debian 10 + to disable apparmor! :
ssh 192.168.100.82 sudo aa-teardown

virsh -c qemu+ssh://192.168.100.82/system blockcommit skillsregistry sda --active --pivot --shallow --verbose

virsh -c qemu+ssh://192.168.100.82/system autostart skillsregistry

(Don't forget to delete the snapshots left behind in case we need to revert!)

Different examples to copy sparse images: