Fixing virt-install on arm64: Security Violation, dnsmasq Port 53, and the qemu:///session Trap

The previous post installed the stack. This one covers what went wrong the first time I used it to create a VM.

The virt-install command in most ARM64 guides is correct. Mine still failed, for four reasons that had nothing to do with the command itself, and everything to do with defaults: libvirt’s, Debian’s, and my shell’s.

Each one produced an error that pointed somewhere other than its cause, which is why they take so long to work out. Taken one at a time, each is quick to fix.

I first hit these in July, on home1, the SD card for my Raspberry Pi 5 where the first VM, lab1, was built. My notes from then record what happened, but not the exact error text, so the original failures are described below in words. Where output is shown, I captured it in October on sbrsl, my other SD card for the same Pi, which has the same software, with the fixes in place.

Checks before you start

Four checks take thirty seconds, and can save much longer:

ls -l /dev/kvm                 # is acceleration available?
virsh version                  # is libvirt answering?
ls /usr/share/AAVMF/           # is the ARM64 UEFI firmware installed?
virsh net-list --all           # is there a network for the VM to join?

The fourth is where my trouble started: it listed no networks at all.

Problem 1: virsh is talking to a different libvirt

The previous post said that installing libvirt-daemon-system defines a default network, and it does. So why did the list come back empty?

virsh connects to libvirt through an address called a URI, and every system has two:

URI	What it is
qemu:///system	The system-wide libvirt, run by the libvirtd service. The default network is here, VM disks go in /var/lib/libvirt/images, and it is what you want.
qemu:///session	A separate libvirt for your user alone, with no system networks and no shared storage. It is meant for desktop VMs that run without special permissions.

They are completely separate. A VM defined in one cannot be seen from the other, and neither can a network. Here is the same question asked of each, on sbrsl:

admin@sbrsl:~ $ virsh --connect qemu:///session net-list --all
 Name   State   Autostart   Persistent
----------------------------------------

admin@sbrsl:~ $ virsh --connect qemu:///system net-list --all
 Name       State    Autostart   Persistent
---------------------------------------------
 default    active   no          yes
 isolated   active   yes         yes

Asked through qemu:///session, there are no networks at all. Asked through qemu:///system, there are two: default, and an isolated network built later in this series. My empty list on home1 was the first of these: virsh was quietly using qemu:///session. Nothing was broken; I was looking in the wrong place.

To see which one virsh uses, ask it:

admin@sbrsl:~ $ virsh uri
qemu:///system

On sbrsl it says qemu:///system, because of the fix below. The fix is to make the system libvirt the default, permanently:

echo "export LIBVIRT_DEFAULT_URI=qemu:///system" >> ~/.bashrc
source ~/.bashrc

Or give it for each command, with --connect qemu:///system, or its short form, -c qemu:///system.

This matters more than it first seems, because the URI decides where a VM lives for its whole life. Create a VM through qemu:///session, and it will not appear in virsh list --all under qemu:///system, will not have the default network, and will keep its disk somewhere else. That shows up later as “my VM disappeared”, which is a much more confusing symptom than an empty table.

That is why the virt-install command in this series includes --connect qemu:///system. Even with the default set, being explicit in the one command that decides a VM’s permanent home is worth the extra line.

Problem 2: the default network will not start

Asking the right libvirt, the default network was there, but inactive. Trying to start it with sudo virsh -c qemu:///system net-start default failed, with an error saying that the address 192.168.122.1 was already in use, on port 53, the DNS port.

The cause was the dnsmasq package from the previous post. Installed on its own, dnsmasq is set up as a system service, and started, listening on port 53 on every interface. libvirt runs its own copy of dnsmasq for the default network, and that copy could not get a port the system service already had.

The fix: libvirt does not need the system service, so stop it and keep it from starting again:

sudo systemctl stop dnsmasq
sudo systemctl disable dnsmasq
sudo virsh -c qemu:///system net-start default
sudo virsh -c qemu:///system net-autostart default

The last line makes the network start whenever the host starts, so this is a one-time fix.

Here is the system dnsmasq on sbrsl, with the fix in place:

admin@sbrsl:~ $ systemctl status dnsmasq
○ dnsmasq.service - dnsmasq - A lightweight DHCP and caching DNS server
     Loaded: loaded (/lib/systemd/system/dnsmasq.service; disabled; preset: enabled)
     Active: inactive (dead)

Two words on the Loaded: line tell the whole story. disabled means it no longer starts with the Pi: the fix. And preset: enabled means Debian’s default for this service is to switch it on, which is exactly why installing the package caused the conflict in the first place.

The general lesson: libvirt manages its own dnsmasq. A system-wide one only gets in its way. Installing dnsmasq-base instead of dnsmasq provides the program without the service, which avoids the problem completely.

(On sbrsl, the default network shows no under Autostart. That setting was turned off at some point during later labs, which caused a problem of its own, in post 5.6.)

Problem 3: the OS variant is not known

admin@sbrsl:~ $ virt-install --name test --os-variant debian12 --memory 512 --disk none --import --print-xml
ERROR    Unknown OS name 'debian12'. See `--osinfo list` for valid values.

I ran that with --print-xml, which only prints the XML virt-install would create, so that I could show the error without creating anything.

--os-variant tells virt-install which operating system is being installed, so that it can choose suitable defaults for the virtual hardware. The valid names come from a database of operating systems, packaged separately, which is often behind the newest releases. Here is what this one knows about Debian:

admin@sbrsl:~ $ virt-install --osinfo list | grep -i debian
debian11, debianbullseye
debian10, debianbuster
debian9, debianstretch
debian8, debianjessie
debian7, debianwheezy
debian6, debian6.0, debiansqueeze
debian5, debian5.0, debianlenny
debian4, debian4.0, debianetch
debian3, debian3.0, debianwoody
debian3.1, debiansarge
debian2.2, debianpotato
debian2.1, debianslink
debian2.0, debianhamm
debian1.3, debianbo
debian1.2, debianrex
debian1.1, debianbuzz
debiantesting

The newest is debian11. The database on Raspberry Pi OS Bookworm is older than Debian 12 itself.

The fix: use the newest name the database knows:

--os-variant debian11

This is safe, and the reason is the useful part. --os-variant only provides hints: which kind of disk and network card suit that operating system, and similar defaults. The operating system actually installed comes from --location. Declaring debian11 while installing Debian 12 gives you Debian 12, with hardware defaults chosen for Debian 11, and both use the same virtio drivers, so nothing differs. That is why post 3.4 finds debian11 in lab1’s XML, while post 3.10 shows it running Debian 12.

--os-variant generic always works as a fallback, with cautious defaults.

Problem 4: Security Violation

This one stops people, because the error appears in the firmware, before any familiar software has run.

The command started, and virt-install downloaded the installer successfully: my notes record linux, 31 MB, and initrd.gz, 38 MB. Then the firmware refused to start it, with a Security Violation, followed by No bootable option.

The download had worked, so the network was fine. The disk had been created. And yet the firmware refused to start what had just been fetched.

Looking at the VM’s XML showed the cause. Its <loader> was /usr/share/AAVMF/AAVMF_CODE.ms.fd, and its <nvram> template was /usr/share/AAVMF/AAVMF_VARS.ms.fd. The .ms in those names means firmware with Microsoft’s Secure Boot keys already enrolled, which enforces Secure Boot: it refuses to run anything not signed by a key it trusts.

Debian’s ARM64 network installer is not signed with those keys. So the firmware did exactly what Secure Boot firmware should do: it refused, and reported a security violation.

The qemu-efi-aarch64 package installs both kinds of firmware, and libvirt’s automatic choice picked the one that enforces Secure Boot:

File	What it is
AAVMF_CODE.fd	Plain UEFI firmware, which does not enforce Secure Boot
AAVMF_CODE.ms.fd	Firmware with Microsoft’s keys, which enforces Secure Boot

Post 3.3 explains why the firmware comes as these pairs of files.

The fix has two parts, and the first is easy to miss.

First, remove the failed VM properly:

sudo virsh -c qemu:///system destroy lab1
sudo virsh -c qemu:///system undefine lab1 --nvram

--nvram is essential. UEFI firmware comes in two parts: read-only code, and a writable file holding its settings, including the boot entries and the Secure Boot state. That settings file is kept separately from the VM’s definition. Remove the VM without --nvram, and the Secure Boot state survives into the next attempt, which then fails in exactly the same way, giving the strong impression that the fix did not work. On a UEFI VM, libvirt actually refuses a plain undefine, and makes you choose --nvram or --keep-nvram, as post 3.7 shows.

Second, create the VM again, naming the plain firmware explicitly:

--boot loader=/usr/share/AAVMF/AAVMF_CODE.fd,loader.readonly=yes,loader.type=pflash,nvram.template=/usr/share/AAVMF/AAVMF_VARS.fd

Read it as four settings:

Setting	Means
loader=	The firmware program: the plain version, without Secure Boot
loader.readonly=yes	The program is shared, read-only, by every VM
loader.type=pflash	It is loaded as flash memory, which is where UEFI expects to be
nvram.template=	The clean template this VM’s own writable settings file is copied from

That one option turned a Security Violation into a working installer. On x86, libvirt’s choice of firmware is usually sensible and this option is rarely needed. On ARM64, with an unsigned installer, it is the line that matters.

A last surprise: the GRUB prompt

One more, which is not a mistake in the command, but looks like a disaster.

The Debian installation finished, the VM restarted, and instead of a login prompt, it stopped at this:

grub>

That is GRUB’s own command prompt. It means GRUB was installed, and the firmware found it and started it, but GRUB did not go on to start Debian. I did not investigate why. A clean restart of the VM was enough:

sudo virsh -c qemu:///system destroy lab1
sudo virsh -c qemu:///system start lab1
sudo virsh -c qemu:///system console lab1

After that, lab1 started Debian normally, and it is the VM all of Part 3 is built on. If the grub> prompt keeps coming back after clean restarts, the bootloader may not be installed where the firmware looks, and post 3.10 shows what that looks like from the firmware’s side.

The checklist

Step	Check	If it fails
1	virsh uri shows qemu:///system	Set LIBVIRT_DEFAULT_URI, or use --connect qemu:///system
2	virsh net-list --all shows default as active	Disable the system dnsmasq, then net-start and net-autostart
3	virt-install --osinfo list has your --os-variant	Use the newest close match, or generic
4	On ARM64, the plain AAVMF_CODE.fd firmware is named with --boot	Remove the failed VM with undefine --nvram, then create it again

The next post goes through the working command, option by option.

Summary

• qemu:///session and qemu:///system are separate libvirts. An empty net-list usually means virsh is using the wrong one. Set LIBVIRT_DEFAULT_URI=qemu:///system, and give --connect qemu:///system to virt-install.
• A system-wide dnsmasq takes port 53 from libvirt’s own copy. Disable it, or install dnsmasq-base instead. Its preset: enabled is why it starts in the first place.
• The OS database can be older than the OS. --os-variant only gives hints, so the newest close match, or generic, is safe.
• On ARM64, libvirt may choose the .ms firmware, which enforces Secure Boot and refuses Debian’s unsigned installer. Name the plain AAVMF_CODE.fd with --boot.
• Use undefine --nvram when removing a failed UEFI VM, or the Secure Boot state carries over to the next attempt.
• A grub> prompt after installation was fixed by a clean restart of the VM.