3.1 · Inside the VM definition

What a libvirt VM Really Is: The XML File Behind Every Tool

KVM · libvirt · Virtualization · Linux

The previous post ended with a guest running, created by one long virt-install command. It also mentioned, almost in passing, that virt-install never talks to QEMU or KVM directly. It generates a document and hands it to libvirt.

Part 3 is about that document. This first post explains what it is, how every libvirt tool relates to it, and a few terms that cause confusion later. The rest of Part 3 reads the document block by block, follows the firmware it boots, and looks at the commands that move a VM between states.

Most people who run libvirt VMs only ever use tools: virt-install to create one, virsh to start and stop it, and perhaps virt-manager to change its settings in a window. The tools work, so the question of what a VM actually is rarely comes up. It is worth answering, because the answer is simple, and it makes the rest of the stack much easier to understand.

A VM is a document

libvirt calls a virtual machine a domain. The word comes from Xen, an older hypervisor, where it meant a running guest. In libvirt, “domain” and “VM” mean the same thing. That is why commands have names like dumpxml, dominfo and domifaddr.

Each domain is described by one XML document. XML is a text format that stores information as nested, labelled elements, such as <memory> or <disk>. libvirt stores the document for a VM called lab1 at /etc/libvirt/qemu/lab1.xml.

To see it, ask libvirt rather than opening the file:

virsh dumpxml lab1

Here is the start of the output for lab1, the Debian guest on my Raspberry Pi 5. The full document is 180 lines, so this is shortened to the lines that matter here:

<domain type='kvm' id='1'>
  <name>lab1</name>
  <uuid>5648e099-729a-4036-b292-bff7a28298c0</uuid>
  ...
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>2</vcpu>
  ...
  <os>
    <type arch='aarch64' machine='virt-7.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/AAVMF/AAVMF_CODE.fd</loader>
    <nvram template='/usr/share/AAVMF/AAVMF_VARS.fd'>/var/lib/libvirt/qemu/nvram/lab1_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
  ...
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/lab1.qcow2' index='1'/>
  ...
</domain>

Even this short extract describes a complete machine. It has 2,097,152 KiB of memory, which is 2 GiB, and two virtual CPUs. It is an ARM64 machine (arch='aarch64'). It boots from UEFI firmware, and its hard disk is a file called lab1.qcow2.

Notice that this is a description, not a set of instructions. The document does not say “allocate 2 GiB of memory”. It says “this machine has 2 GiB of memory”. Turning that description into a running machine is libvirt’s job.

Why virsh dumpxml and not the file itself

The file in /etc/libvirt/qemu/ belongs to libvirt. libvirt does not watch it for changes, so if you edit it by hand while libvirt is running, nothing happens, and libvirt may later overwrite your edit. The safe way to read the document is virsh dumpxml, and the safe way to change it is virsh edit, which opens it in your text editor and checks your changes before saving them.

What the document points to

The document is the definition of the VM, but it does not contain the VM’s data. Instead, it points to files that do. For lab1, there are two:

FileWhat it holdsWhere the XML points to it
/var/lib/libvirt/images/lab1.qcow2The virtual hard disk: the operating system and everything installed on it<source file=...> inside <disk>
/var/lib/libvirt/qemu/nvram/lab1_VARS.fdThe firmware’s settings, including its list of boot entries<nvram> inside <os>

The second file is easy to overlook. Every VM that boots with UEFI firmware has one, and on ARM64 that means every VM. Post 3.3 explains why the firmware comes as two files, and why that matters.

So a more precise answer to “what is a VM?” is: one XML document, plus the files it points to. There is no hidden database behind it. If you want to know what a VM is, read its XML. If you want to change it, change its XML. If you want to move it to another machine, copy the XML and the files it points to. (Snapshots add records of their own, which Part 4 covers.)

Where the tools fit

Once you know the document exists, every libvirt tool can be described by what it does to it:

   virt-install ─────────► writes a new document from command-line options
   virt-manager ─────────► a graphical program that reads and writes the document
   virsh edit ───────────► opens the document in your text editor
   virsh dumpxml ────────► prints the document
   virsh define ─────────► registers a document as a domain
   virsh start ──────────► reads the document and builds a running machine from it

virt-install, which most people meet first, is a generator. It turns its options into an XML document, hands the document to libvirt, and starts the result. Everything it does could also be done by writing the XML by hand and running virsh define. It exists because writing that document from scratch is tedious, not because it can do something the document cannot express.

This reverses the usual way of thinking about it. The tools are not the real interface, with the XML as a hidden detail. The XML is the real interface, and the tools are conveniences built on top of it.

Persistent and transient domains

libvirt has two kinds of domain, and the difference is simply whether the document is saved:

  • A persistent domain has its XML stored on disk. It continues to exist as a definition when it is not running, and after the host restarts. lab1 is persistent, which is why it still appears in virsh list --all after it has been shut down.
  • A transient domain exists only while it is running. Its XML is never stored, so when it stops, it disappears completely.

virt-install creates persistent domains. Transient domains are less common when you work by hand, but automated systems use them to start machines on demand without leaving definitions behind.

define, create and start

These three commands are often confused, and the difference follows directly from persistent and transient domains:

CommandWhat it does
virsh define lab1.xmlRegisters a persistent domain from an XML file. It saves the definition, but does not start anything.
virsh create lab1.xmlStarts a transient domain directly from an XML file. It runs the machine now, and saves nothing.
virsh start lab1Starts a domain that libvirt already knows about, by its name.

In short, define means “remember this machine”, create means “run this machine now, without remembering it”, and start means “run the machine you already remember”.

The confusing one is create. It sounds like the command for making a new VM, but it actually runs a VM that libvirt has not saved. The command that makes a VM, in the sense most people mean, is define.

The opposites follow the same pattern, and one of them has an unfortunate name. The opposite of define is undefine, which deletes the saved definition. The opposite of create is destroy, which stops a running machine immediately, like pulling its power cable. Despite its name, virsh destroy deletes nothing: the XML, the disk and the firmware settings all remain. Post 3.7 looks at this in more detail.

So there are two separate questions, each with its own pair of commands:

QuestionCommands
Does a saved definition exist?define and undefine
Is the machine running right now?create and destroy (and start for a defined machine)

What libvirt turns the document into

There is one more fact that makes the whole document make sense: libvirt does not run virtual machines itself.

When you start a domain, libvirt reads its XML, translates it into a very long qemu-system-aarch64 command, and runs that command. The QEMU process that results is the virtual machine. libvirt keeps a connection to that process so that it can control it and report its state.

   domain XML  ──libvirt translates──►  qemu-system-aarch64 -machine virt ...
                                                 │
                                                 ▼
                                      the QEMU process is the VM

This means every element in the document corresponds to something QEMU can be told on its command line. <memory> becomes the -m option, and <vcpu> becomes -smp. A <disk> becomes a set of -blockdev options, which describe the file and its format, and a -device option, which describes the virtual disk controller the guest sees. The XML is an easier, descriptive way of writing that command, plus a long list of sensible defaults you would otherwise have to write yourself.

libvirt can show you the translation:

sudo virsh domxml-to-native qemu-argv /etc/libvirt/qemu/lab1.xml

For lab1, the output is a single line of several thousand characters, and every option in it can be traced back to a block in the document. Post 3.9 works through that translation, and then starts the VM from a QEMU command written by hand, with libvirt out of the loop.

Why it pays to read the XML

The practical reason to learn this document, rather than staying with the tools, is that every new capability appears as something in the document.

Adding a second disk is a new <disk> block. Moving a VM from the default NAT network to your real network is a change to its <interface> block. Pinning virtual CPUs to particular physical cores is a <cputune> block. Giving a guest a real PCI device from the host is a <hostdev> block.

So the question “how do I do this with a libvirt VM?” usually becomes “which element does this, and where does it go?”. That is easier to answer than searching for the right tool option, and the answer is the same whether you use virsh, virt-manager, Terraform, or a Python script that talks to libvirt directly. They all end up writing the same document.

Summary

  • libvirt calls a VM a domain. Each domain is defined by one XML document, which libvirt stores in /etc/libvirt/qemu/.
  • The document points to the files that hold the VM’s data: its disk image and, for UEFI VMs (which on ARM64 means every VM), its firmware settings file.
  • Read the document with virsh dumpxml, and change it with virsh edit. libvirt does not notice edits made directly to the file.
  • Every tool, including virt-install, virt-manager and virsh, works by reading or writing this document.
  • A persistent domain has a saved definition. A transient domain exists only while it runs.
  • define saves a definition without starting it, create runs a VM without saving it, and start runs a VM libvirt already knows. destroy stops a running VM and deletes nothing.
  • libvirt translates the document into a qemu-system-aarch64 command, and the resulting QEMU process is the VM.

The next post steps back from libvirt to something more basic: what happens when any computer, real or virtual, is switched on, before an operating system exists.

Comments

get new posts

About one email a week, and only when there is something new.

Subscribe →