Documentation

WASPA / ISPA redactions

Jeff — Tue, 22 Aug 2023 06:42:45 +0000

WASPA / ISPA redactions Jeff Tue, 22/08/2023 - 08:42

Trim Mailman archive for data deletion policy

Jeff — Thu, 06 Jul 2023 06:40:33 +0000

Trim Mailman archive for data deletion policy Jeff Thu, 06/07/2023 - 08:40

Resize qcow2 with blockresize on running VM

Admin — Wed, 09 Mar 2022 03:46:15 +0000

Resize qcow2 with blockresize on running VM Admin Wed, 09/03/2022 - 05:46

So I sent an email to libvirt-users@redhat.com (there are some amazing wizards there!) and got an almost immediate response from Peter:

You can resize a QCOW2 (or any VM disk image) with virsh blockresize VM /path/to/image.qcow2 SIZE'.

This requires the VM to be running.

virsh blockresize ISPA /var/lib/libvirt/images/ispa.qcow2 300G

Result:
jeff@ispa-mx ~ $ sudo fdisk -l [sudo] password for jeff: Disk /dev/sda: 300 GiB, 322122547200 bytes, 629145600 sectors

Next, Install gdisk and cloud utils package for growpart, a Linux command line tool used to extend a partition in a partition table to fill available space.

apt install cloud-utils gdisk

Problem is that often/usually, in our case, the swap partition sits at the end of the disk, so it needs to be removed and added back after growing the primary data partition sda1.

root@ispa-mx:~# lsblk 
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0   300G  0 disk 
├─sda1   8:1    0 212.2G  0 part /
└─sda2   8:2    0   7.8G  0 part [SWAP]

Turn swap OFF.

swapoff -a

EDIT /etc/fstab and comment it OUT as a reboot may be required and it would hang the system because after recreating the partition the UUID will have changed!

Overview to end: Run fdisk, delete the swap, then recreate the swap partition at the end of the disk where it belongs. Write fdisk changes and run partprobe to get the kernel reinitialised with the change.

Then:

growpart /dev/sda 1 #Note the space

resize2fs /dev/sda1

Then fix the swap partition:

mkswap /dev/sda2 

Setting up swapspace version 1, size = 7.8 GiB (8389423104 bytes)
no label, UUID=5a00a73e-a01c-4900-8a63-ae76c42fa61f

Turn swap back on:

swapon -a

Note the new UUID. Edit /etc/fstab and update the swap UUID.

Fin.

Category

Documentation

Tags

KVM

qemu-kvm

swap

fdisk

partprobe

growpart

resize2fs

ECC and certbot automated renewal

Admin — Sun, 21 Jun 2020 09:34:17 +0000

ECC and certbot automated renewal Admin Sun, 21/06/2020 - 11:34

The problem is that the certbot program cannot renew a certificate for an ECC public key.

Instead of running certbot renew, we should roll our own.

Take care to rename the Exim and Dovecot certificates in the appropriate place.

This works:

root@abispa ~/certbot-renewal $ cat renew-certbot.sh 
#!/bin/sh

LOGFILE=/root/certbot-renewal/certbot-renewal.log
ARCHIVE=/etc/letsencrypt/ecc-archive/abispa.waspa.org.za

## "Renew" the RSA certificate -- this actually generates a fresh
## 4096-bit RSA key pair and creates a certificate from the public key.
echo "RSA renewal ====================================" > $LOGFILE
certbot --force-renewal --rsa-key-size 4096 renew >> $LOGFILE 2>&1

## Use the Certificate-Signing Request for the existing ECC
## public key, and request a new certificate.
## You can read a CSR with:
## $ openssl req -noout -text -in /path/to/csr.pem
echo "ECC renewal ====================================" >> $LOGFILE
certbot certonly --non-interactive --apache \
	-d abispa.waspa.org.za \
	--email ops@ff.co.za \
	--csr /etc/letsencrypt/csr/ecc-csr.pem \
	--agree-tos >> $LOGFILE 2>&1

## The above creates three files in the local directory.
## Move them into place.
echo "Installing files ===============================" >> $LOGFILE
mv -fv 0000_cert.pem  $ARCHIVE/cert.pem >> $LOGFILE
mv -fv 0000_chain.pem $ARCHIVE/chain.pem >> $LOGFILE
mv -fv 0001_chain.pem $ARCHIVE/fullchain.pem >> $LOGFILE

## Fix Debian-exim group permissions.
## Remember to update dovecot & exim4 cert locations!
chmod 755 -R /etc/letsencrypt/{ecc-archive,ecc-live}; chgrp -R Debian-exim /etc/letsencrypt/{ecc-archive,ecc-live}
chmod 755 -R /etc/letsencrypt/{rsa-archive,rsa-live}; chgrp -R Debian-exim /etc/letsencrypt/{rsa-archive,rsa-live}

## Restart mail & web server so they use the new certs.
echo "Apache restart =================================" >> $LOGFILE
service apache2 stop >> $LOGFILE 2>&1
service apache2 start >> $LOGFILE 2>&1 
echo "Dovecot restart =================================" >> $LOGFILE
service dovecot stop >> $LOGFILE 2>&1
service dovecot start >> $LOGFILE 2>&1 
echo "Exim4 restart =================================" >> $LOGFILE
service apache2 stop >> $LOGFILE 2>&1
service apache2 start >> $LOGFILE 2>&1

Category

Tags

Elliptic Curve Crypto & LetsEncrypt

Admin — Sun, 21 Jun 2020 08:53:23 +0000

Elliptic Curve Crypto & LetsEncrypt Admin Sun, 21/06/2020 - 10:53

Migrate a VM with Minimal Downtime

Admin — Tue, 26 Nov 2019 08:17:45 +0000

Migrate a VM with Minimal Downtime Admin Tue, 26/11/2019 - 10:17

Migrate a VM with Minimal Downtime

This script takes a snaphot of the running VM, which leaves the qcow2 backing images in a known state. All VM activity is then recorded in the snapshots, while the backing images are rsync'd to a reciprocal hypervisor on the 1G VLAN.

After the backing images are transferred, the VM is shutdown and snapshots copied to the reciprocal hypervisor. The server is defined on the new host, started, and the snapshots are merged into the backing image/s.

Total downtime depends on the size of the snapshots, but is usually less than a minute or two. The whole process takes about an hour an a half for a ~200GB QCOW2 image over a 1GB/sec VLAN.

Example 1:

jeff@sheri:/var/lib/libvirt/images/scripts$ cat skillsregistry.migrate.sh 
#!/bin/bash
################################################################################
######################  Migrate VM to a new hypervisor  ########################
# Authors: Jeff Brown / Lucio de Re
# Date: 2019/11/26
#
# This script takes a snaphot of the running VM, which leaves the qcow2 backing
# images in a known state. All VM activity is then recorded in the snapshots,
# while the backing images are rsync'd to a reciprocal hypervisor on the VLAN.
# After the backing images are transferred, the VM is shutdown and snapshots
# copied to the reciprocal hypervisor. The server is defined on the new host,
# started, and the snapshots are  merged into the backing image/s.
# Total downtime depends on the size of the snapshots, but is usually
# less than a minute or two. The whole process takes about an hour and a half.
#
# NB. This script must be run by a user who is a member of libvirt-qemu group.
# NB. The user must also be able to run sudo aa-teardown/chown/chmod without password.
# E.g. jeff ALL=NOPASSWD:/bin/chown, /bin/chmod
#
# 192.100.0.4 (vespa.org.za)
# 192.100.0.82 (abispa.org.za)
#
################################################################################

cd /var/lib/libvirt/images/
virsh snapshot-create-as --domain skillsregistry skillsregistry.snapshot --diskspec sda,file=/var/lib/libvirt/images/snapshots/skillsregistry.snapshot.qcow2 --disk-only --atomic
export RSYNC_PASSWORD="real password for rsyncd secret"
rsync -ShavW --progress /var/lib/libvirt/images/skillsregistry.qcow2 rsync://tergum@192.168.100.82/bvar/lib/libvirt/images/skillsregistry.qcow2
virsh shutdown skillsregistry

x=0
while sleep 5; do
virsh list --all | grep skillsregistry | grep "shut off" && break
x=$(expr $x + 1)
[ $x -gt 120 ] && { echo >&2 timed out after $(expr $x \* 5) seconds; exit 3; }
done

virsh autostart skillsregistry --disable
virsh -c qemu:///system dumpxml skillsregistry > /var/lib/libvirt/images/xml/skillsregistry.xml
scp /var/lib/libvirt/images/xml/skillsregistry.xml 192.168.100.82:/var/lib/libvirt/images/xml/
sudo find . -type f -exec chown libvirt-qemu:libvirt-qemu {} \; -exec chmod g+rw {} \;
rsync -ShavW --progress /var/lib/libvirt/images/snapshots/skillsregistry.snapshot.qcow2 rsync://tergum@192.168.100.82/bvar/lib/libvirt/images/snapshots/

export RSYNC_PASSWORD="nopass"

virsh -c qemu+ssh://192.168.100.82/system define /var/lib/libvirt/images/xml/skillsregistry.xml
virsh -c qemu+ssh://192.168.100.82/system start skillsregistry

x=0
while sleep 5; do
virsh -c qemu+ssh://192.168.100.82/system list --all | grep skillsregistry | grep "running" && break
x=$(expr $x + 1)
[ $x -gt 120 ] && { echo >&2 timed out after $(expr $x \* 5) seconds; exit 3; }
done

# Required for Debian 10 + to disable apparmor! :
ssh 192.168.100.82 sudo aa-teardown

virsh -c qemu+ssh://192.168.100.82/system blockcommit skillsregistry sda --active --pivot --shallow --verbose

virsh -c qemu+ssh://192.168.100.82/system autostart skillsregistry

(Don't forget to delete the snapshots left behind in case we need to revert!)

Different examples to copy sparse images:

rsync --rsync-path="ionice -n 7 nice -n19 rsync" -havW --progress /var/lib/libvirt/images/UAT.qcow2 192.168.100.1:/var/lib/libvirt/images/

cd /var/lib/libvirt/images/; tar czf - DNC.qcow2 | ssh 192.168.100.1 cd /var/lib/libvirt/images/ \&\& tar xzvf -

rsync -SWv --show-progress rsync://tergum@192.168.100.4/bvar/lib/libvirt/images/FFDev.qcow2 /tmp/

Category

Documentation

Tags

qemu-kvm

Install QEMU-KVM on Debian 9

Jeff — Mon, 25 Feb 2019 07:17:07 +0000

Install QEMU-KVM on Debian 9 Jeff Mon, 25/02/2019 - 09:17

Allow users to run virsh and manage VMs

Admin — Tue, 13 Nov 2018 04:59:46 +0000

Allow users to run virsh and manage VMs Admin Tue, 13/11/2018 - 06:59

https://libvirt.org/aclpolkit.html
https://major.io/2015/04/11/run-virsh-and-access-libvirt-as-a-regular-u…
https://www.poftut.com/use-virt-manager-libvirt-normal-user-without-roo…

Defining custom rules requires creation of a file in the /etc/polkit-1/rules.d directory with a name chosen by the administrator (100-libvirt-acl.rules would be a reasonable choice). See the polkit(8) manual page for a description of how to write these files in general.

We're going to allow members of the libvirt group to manage VM's:

mkdir /etc/polkit-1/rules.d
vi /etc/polkit-1/rules.d/100-libvirt-acl.rules

polkit.addRule(function(action, subject) { if (action.id == "org.libvirt.unix.manage" && subject.local && subject.active && subject.isInGroup("libvirt")) { return polkit.Result.YES; } });

NB. Add to user's .bashrc:

export LIBVIRT_DEFAULT_URI=qemu:///system

Add bash completion library:

Download the bash completion library: virsh_bash_completion

How to use it: Copy virsh_bash_completion file to /etc/bash_completion.d/ then restart bash.

(Source)

Category

Tags

Backup Virtual Machine

Jeff — Sun, 11 Nov 2018 08:21:51 +0000

Backup Virtual Machine Jeff Sun, 11/11/2018 - 10:21

Sources:

1. As root, locate the virtual machine you want to back up using virsh:

virsh -c qemu:///system list # shows only running

virsh list --all # shows all

2. Dump the configuration for your virtual machine:

virsh -c qemu:///system dumpxml MACHINE_NAME > /var/lib/libvirt/images/safe/MACHINE_NAME.datestamp.xml

3. You may need to locate the image directory:

virsh -c qemu:///system domblklist MACHINE_NAME

4. Now prepare the backup of this machine using a snapshot:

virsh snapshot-create-as --domain MACHINE_NAME IMAGE_NAME.snapshot --diskspec sda,file=/var/lib/libvirt/images/safe/IMAGE_NAME.snapshot.qcow2 --disk-only --atomic

Note: If you get an error, then the disk is not converted to virtio-scsi, and should be vda, not sda.

After creating the snapshot, all disk operations from the guest will be directed to it so that you can copy the original disk image to a safe place.

You can establish this by running, as example:

virsh -c qemu:///system domblklist ispa

Target     Source
------------------------------------------------
hda        -
sda        /var/lib/libvirt/images/safe/ispa.snapshot.qcow2

5. Copy to external server and include both the qcow2 disk image and the machine configuration (the XML file):

su - yourusername # you must be in libvirt-qemu group

rsync -hav --progress /var/lib/libvirt/images/IMAGE_NAME.qcow2 192.168.100.2:backups/ # IP address of prionyx on 1GB VLAN

6. Merge snapshot:

After the file transfer is completed, you should merge changes written to snapshot, back to original disk image. This operation is called 'active block commit':

virsh blockcommit MACHINE_NAME sda --active --pivot --shallow --verbose

Check block devices again. If the block commit completed successfuly your virtual machine should use its original image again.

virsh -c qemu:///system domblklist MACHINE_NAME

Example Backup script for running VMs:

jeff@sheri:~$ cat /var/lib/libvirt/images/scripts/ispa.backup.sh 
#!/bin/bash
################################################################################
#
# This script takes a snaphot of the running VM, which leaves the main qcow2
# image in a known state. All VM activity is then recorded in the snapshot,
# while the main qcow2 image can be rsync'd to a remote backup server.
# After the image has been transferred, the snapshot is merged back to the
# main image, and then deleted.
#
# NB. This script must be run by a user who is a member of libvirt-qemu group.
#
################################################################################


virsh -c qemu:///system dumpxml ISPA > /var/lib/libvirt/images/xml/ispa.xml
scp /var/lib/libvirt/images/xml/ispa.xml 192.168.100.4:/var/lib/libvirt/images/xml/
virsh snapshot-create-as --domain ISPA ISPA.snapshot --diskspec sda,file=/var/lib/libvirt/images/snapshots/ispa.snapshot.qcow2 --disk-only --atomic
rsync --rsync-path="ionice -c 3 nice rsync" -havW --progress /var/lib/libvirt/images/ispa.qcow2 192.168.100.4:/var/lib/libvirt/images/sheriff/
virsh blockcommit ISPA sda --active --pivot --shallow --verbose
virsh snapshot-delete ISPA --metadata ISPA.snapshot
rm -rf /var/lib/libvirt/images/snapshots/ispa.snapshot.qcow2
#rm /var/lib/libvirt/images/xml/ispa.xml

Category

Tags

Reclaiming space on qcow2 disk image

Jeff — Sat, 01 Sep 2018 08:20:52 +0000

Reclaiming space on qcow2 disk image Jeff Sat, 01/09/2018 - 10:20

Update 2018-11-11:

Source: dustymabe.com/2013/06/11/recover-space-from-vm-disk-images-by-using-discardfstrim/

Recommended method is to use fstrim. See: pve.proxmox.com/wiki/Shrink_Qcow2_Disk_Files

"The recommended version is to pass TRIM commands (known from SSDs) from the VM to the backing storage. This has the advantage that it works automatically, does not need to write the whole free parts of all disks to zero and must only be setup once."

The alternative manual method outlined below, is slow and painful - but seemingly necessary for the occasional apparent failure of fstrim, as experienced first-hand.

Convert the storage 'driver' from virtio-blck to virtio-scsi

Use VMM (Virtual Machine Manager) in a vncserver session on the hypervisor, over the VPN.

NB: Shut the VM down, first!

Remove the current virtio-blk disk image:

And add it back as virtio-scsi:

Then, to enable TRIM in the guest, first edit the VM's XML configuration file on the hypervisor:

virsh edit MACHINE_NAME

And add discard='unmap' to the driver.

  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' discard='unmap'/>
      <source file='/var/lib/libvirt/images/new-ff.qcow2'/>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>

Start the VM and TRIM

virsh start new-ff

Then, SSH to the VM, and as root, run the following command:

fstrim -v /

Add to root's crontab to perform a trim weekly.

crontab -e

# Trim filesystem daily to recover space
0 2 * * 1      /sbin/fstrim -v / >> /home/jeff/fstrim.log

Alternative method:

Shutdown and backup the VM's qcow2 disk image.

Mount the qcow2 image using this guide.

Zero-fill the drive (dd if=/dev/zero of=/some/file until you run out of space)
delete /some/file

qemu-img convert -O qcow2 original_image.qcow2 deduplicated_image.qcow2

Or...

In a terminal on the VM, run the dd if=zero commands until you run out of disk space. Before running this, be sure to stop any applications running on the guest otherwise errors may result.

Category

Tags