KVM vs QEMU vs libvirt: Who Does What

The previous post covered the processor’s virtualization extension, and how KVM uses it.

“What is the difference between KVM and QEMU?” is one of the most common questions in Linux virtualization, and most answers stack three definitions on top of each other. Definitions are easily forgotten. A better way is to follow a chain of causes: each piece exists to fill a specific gap that the piece below it leaves. So instead of defining everything at once, this post starts with what KVM deliberately does not do, and follows the consequences upwards.

KVM handles the processor and memory, and nothing else

The previous post covered what KVM contributes. It uses the processor’s virtualization extension so that guest code runs directly on real cores, trapping only on privileged operations. And it manages the guest’s memory, mapping the guest’s idea of physical addresses onto real memory in the host, and keeping each guest confined to its own.

KVM does both very efficiently. But that is all it does.

A processor and some memory do not make a computer. A virtual machine also needs a motherboard, a PCI bus, a keyboard, a display, serial ports and disk controllers, and KVM deliberately includes none of them. With KVM alone, a VM could not start: it would have no disk to read an operating system from, and nowhere to show it.

That is not an oversight. It is a design decision, and post 1.1 explained why: because the hypervisor is part of a full Linux kernel, it can rely on everything else Linux already does well. Building imitation devices into KVM would duplicate work that other parts of the system already handle.

So KVM leaves an obvious gap. A guest kernel starts, looks for a disk, and finds nothing. Something has to answer.

QEMU fills the gap: it imitates the devices

QEMU, the Quick Emulator, is an ordinary program that imitates a whole machine’s worth of devices. When the guest looks for a disk controller, QEMU answers. When the guest writes a character to a serial port, QEMU receives it. When the guest sends a network packet, QEMU passes it to the host’s network, or, with vhost-net, arranges for the kernel to do that directly, as post 5.11 shows.

That shows exactly where KVM hands over to QEMU. When the guest tries to use a device, the processor traps, as the previous post described, and KVM, which does not know how to be a device, pauses the guest and hands the request up to QEMU. Inside QEMU, a piece of C code imitates the chip the guest expects, deals with the request, and tells KVM to let the guest carry on. Every piece of “hardware” the guest sees is imitated in software. Post 1.4 shows how that works, since “imitates the devices” covers a lot.

QEMU can imitate a processor too. On its own, it is a complete machine emulator: without KVM, it translates the guest’s instructions in software, and starts the operating system anyway, slowly. Paired with KVM, it hands the processor and memory to KVM, and keeps only the devices.

That pairing is what a running VM is:

Two facts follow from this, and they shape how VMs behave on a real Linux system.

One VM is one QEMU process. A virtual machine is not a mysterious container. It is one ordinary Linux process, running a program such as qemu-system-aarch64, for ARM64 guests, or qemu-system-x86_64, for x86 ones. It appears in ps and top like any other program, and if the process ends, the VM is gone at once. Post 5.11 finds exactly that process for a running guest on my Raspberry Pi.

Inside it, each virtual processor is an ordinary thread. A guest with four virtual processors has four of them, each running a loop: run guest code, stop when something traps, deal with it, carry on. Because these are ordinary threads, the Linux scheduler manages them like any others, which is why a VM’s virtual processors can be tied to particular physical cores with ordinary Linux tools.

The two halves hand over to each other constantly. KVM runs guest code at full speed until the guest uses a device, such as writing to a disk or sending a packet. Then it pauses that virtual processor’s thread and hands control to QEMU, whose imitation of the device responds, updates its state, and tells KVM to carry on. Post 1.5 looks at that handover.

So KVM and QEMU together make a complete machine. Which raises the next question: how do you start one?

libvirt: from a long command to a lasting document

QEMU has one problem as something to use directly: describing a VM means giving every detail of its hardware as an option on its command line.

To start an ordinary VM, you have to give each disk’s format and bus, each network card’s model and how it connects to the host, the firmware files, the processor model, the serial console, and how the memory is laid out. A realistic command easily runs to dozens of options. For the Debian VM built in Part 2, the command libvirt generated had 57 options, and was 4,784 characters long, as post 3.9 shows.

Nobody wants to type that, and nobody wants to remember it.

libvirt solves this by making the VM a document rather than a command. You describe the machine once, in a structured XML document, and the libvirt service, libvirtd, reads it, builds the long QEMU command from it, starts the process, and keeps track of it from then on.

More than starting VMs

libvirt does more than start processes:

• It manages the things VMs depend on: virtual networks, storage pools and snapshots.
• It offers a stable interface, so tools written for libvirt keep working across new versions of QEMU.
• It keeps a control connection to each running QEMU, called the monitor. That is how a disk can be added to a running VM, or a snapshot taken, without restarting it.

The tools you actually type or click all talk to libvirtd:

Tool	What it is
virsh	The command-line tool, used for most of this series
virt-manager	A desktop program
virt-install	A tool for creating new VMs
Cockpit	A web interface for managing servers

None of them talks to QEMU directly. They all ask libvirtd, which does the work.

The result is that a VM is its XML document, plus the files it points to: its disk and, for UEFI VMs, its firmware settings. Change a line in the document, and you change the machine. That document is the centre of all libvirt work, and Part 3 begins with it, in post 3.1.

The three sentences

Reduced to the essentials:

• KVM runs the guest’s processor and memory on real hardware, at full speed.
• QEMU imitates all the devices the guest believes it has.
• libvirt keeps each VM as an XML document, and runs QEMU for you.

But the chain of causes is more useful than the definitions, because it explains why the stack has this shape. KVM is small, because Linux already provides everything except the virtualization of the processor and memory. That leaves the devices, so QEMU imitates them, in an ordinary program. QEMU’s command line is impractical to write by hand, so libvirt turns it into a lasting document, and manages it.

To test the idea, take one piece away, and see what happens:

Without	What happens
libvirt	Everything still runs, fast, with all its devices, but you write and maintain enormous command lines by hand
QEMU	The VM has a fast processor and memory, but no motherboard, network card or disk to start from
KVM	Everything still works, with all its devices, but slowly, entirely in software

What this means for the rest of the series

This division of work is a map for everything that follows.

• Storage, in Part 4: disk formats, snapshots and thin provisioning are all about what QEMU’s virtual disk points to on the host.
• Networking, in Part 5: all about what QEMU’s virtual network card plugs into, whether a NAT network, a bridge on your real network, or an isolated one.
• Performance crosses the boundary. Replacing imitated devices with virtio ones happens on QEMU’s side. Tying virtual processors to physical cores happens in the host’s scheduler, and is only possible because they are ordinary threads.

Each of these changes one part of a machine whose overall shape is now clear.

Summary

• KVM handles the guest’s processor and memory only. It imitates no devices, by design, because Linux already provides everything else.
• QEMU fills that gap: an ordinary program that imitates disks, network cards, consoles and firmware. It can imitate a processor too, but hands that job to KVM when it can.
• One running VM is one QEMU process, with one thread for each virtual processor.
• libvirt exists because QEMU’s command line is impractical to write by hand. It keeps each VM as XML, builds the command, starts QEMU, and controls it through the monitor.
• virsh, virt-manager, virt-install and Cockpit all talk to libvirtd.
• A VM is its XML document, plus the files it points to.