1.5 · How the stack works
What ioctl(KVM_RUN) Does: The VM Exit Loop Explained
The previous article established that guest code runs natively until it touches a device, at which point control has to reach QEMU.
That handoff has been asserted twice now without being opened up. This article opens it.
ioctl(vcpu_fd, KVM_RUN) is the single most important line in the entire stack. Everything else — libvirt, the XML, the device models, VirtIO — is scaffolding around this one call, looped forever. Understanding it precisely is what turns “virtualization is slow at I/O” from a rule you’ve memorized into a cost you can predict.
We’ll build up to it, because the call only makes sense once you know what a file descriptor is and how ioctl works generally.
File descriptors, briefly
A file descriptor is a non-negative integer that identifies something a process has open. That’s the whole concept.
Every process has a private table maintained by the kernel. When the process opens something, the kernel creates an entry pointing at the underlying kernel object and returns the table index. From then on, the process refers to that thing by its number.
[ Your Program ] --( reads/writes via integer: 3 )--> [ FD Table ] --> [ actual file / device ]Three are opened automatically for every process: 0 is standard input, 1 standard output, 2 standard error. Anything you open next gets the lowest free integer, usually starting at 3.
The reason this generalizes so well is Linux’s “everything is a file” design. A regular file, a directory, a network socket, a pipe, a hardware device — all are reached through file descriptors, using the same read(), write(), and close() calls. The kernel dispatches to the right implementation based on what the descriptor points at.
ioctl: the escape hatch for everything read and write can’t express
read() and write() cover moving bytes. They don’t cover much else.
You cannot “write” to a CD drive to make it eject. There’s no meaningful way to read() a serial port’s baud rate, or to express “change it to 115200” as a byte stream. These are control operations on a device, not data transfers.
Unix’s answer is ioctl — I/O control — a deliberately generic system call meaning “send an arbitrary command to whatever this descriptor points at.”
int ioctl(int fd, unsigned long request, ...); ioctl(fd, REQUEST, argument)
│ │ │
│ │ └─ pointer to a struct (data in and/or out)
│ └────────── which command (a number)
└──────────────── which open thing to talk toThe kernel side of each device decides what each command number means. It’s the extension point that lets any driver expose operations the standard file interface never anticipated. Terminals use it for window size. Network interfaces use it for addresses. And KVM uses it for everything.
KVM’s three-level ladder of descriptors
KVM exposes itself as /dev/kvm, a character device. QEMU opens it and gets a descriptor. That descriptor plus a stream of ioctl calls is the entire QEMU-to-KVM interface. There’s no other channel.
The structure that makes this workable is that there isn’t one descriptor — there are three levels, each created by an ioctl on the level above:
open("/dev/kvm") ───────────────► the SYSTEM fd
│ (KVM as a whole)
│ ioctl(KVM_CREATE_VM)
▼
a VM fd ──────────────────────────► one virtual machine
│ (its memory map, its devices)
│ ioctl(KVM_CREATE_VCPU)
▼
a vCPU fd ────────────────────────► one virtual CPU
(THIS is what KVM_RUN runs)Creating a VM is therefore a descent down this ladder:
KVM_CREATE_VMon the system fd — “give me an empty machine.” Returns a VM fd.KVM_SET_USER_MEMORY_REGIONon the VM fd — “here’s memory I allocated; treat it as the guest’s physical RAM.” QEMU allocates guest memory as an ordinarymmapregion in its own address space and registers it with KVM. This is the sandbox boundary: the guest can only address what was registered.KVM_CREATE_VCPUon the VM fd — “add a CPU.” Returns a vCPU fd.KVM_RUNon the vCPU fd — “run this CPU now.”
Each vCPU thread inside QEMU holds one vCPU fd and loops on KVM_RUN against it. A four-vCPU guest is four threads, four descriptors, four independent loops.
What KVM_RUN actually does
This is the part worth slowing down for, because the call does not behave like a normal function call.
When a QEMU vCPU thread calls ioctl(vcpu_fd, KVM_RUN), the thread enters the guest and does not return until the guest does something KVM cannot handle alone.
┌─ QEMU vCPU thread, running normal host code
│
│ calls ioctl(vcpu_fd, KVM_RUN)
│ │
│ ▼
│ ── enters the KERNEL (KVM) ──
│ │
│ │ KVM loads the guest's saved CPU state
│ │ (registers, program counter) into the
│ │ REAL core, switches to guest mode
│ │ via the EL2 extension
│ ▼
│ ════ THE REAL CORE IS NOW RUNNING GUEST CODE ════
│ │
│ │ guest runs directly, full speed:
│ │ arithmetic, branches, memory — all native.
│ │ KVM is NOT involved instruction-by-instruction.
│ │ the ioctl call has "not returned yet."
│ │
│ │ ...until the guest does something privileged
│ ▼
│ ════ VM EXIT ════ CPU traps back into KVM
│ │
│ │ KVM saves guest CPU state, switches back
│ │ to host mode, examines WHY it exited
│ │
│ ┌────┴─────────────────────────┐
│ │ │
│ KVM can handle it KVM cannot — it's a
│ itself (timer, in-kernel device QEMU emulates
│ interrupt controller) │
│ │ │
│ │ re-enters the guest │ the ioctl FINALLY
│ │ immediately — QEMU │ RETURNS to QEMU
│ │ never even wakes ▼
│ │ back in QEMU's C code
└───┘ with an exit reason
│
▼
QEMU's device model handles it,
then calls ioctl(KVM_RUN) againThree things in there deserve to be stated plainly.
The call blocks for a long time, on purpose. From QEMU’s perspective, it called a function and that function didn’t return for millions of guest instructions. The entire time the guest is computing, the QEMU thread is parked inside a single ioctl. This is elegant rather than wasteful — QEMU isn’t polling or supervising anything. It’s asleep in a system call, woken only when there’s work for it.
Guest CPU state has to be saved and restored across the boundary. Entering the guest means loading the guest’s registers into the real core; exiting means saving them back out. That’s not free, and it’s part of why an exit costs what it does.
Not every VM exit returns to QEMU. This is the subtlety that most descriptions omit, and it matters.
Cheap exits and expensive exits
There are two tiers of VM exit, and conflating them makes performance behaviour look arbitrary.
Some exits KVM handles entirely in the kernel and then immediately re-enters the guest, without the ioctl ever returning. QEMU stays asleep and never learns it happened. Timer operations qualify. So do accesses to devices KVM emulates in-kernel — on ARM64, most importantly the GIC, the interrupt controller. Interrupt handling is far too frequent to route through userspace, so KVM handles it directly.
Other exits KVM cannot resolve, because the guest touched a device that exists only as a QEMU device model. Now the ioctl genuinely returns, QEMU wakes up, runs the relevant C function, and calls KVM_RUN again.
CHEAP — guest traps → KVM handles it → straight back into the guest
(stays in kernel; QEMU never wakes)
EXPENSIVE — guest traps → KVM can't handle it → ioctl returns →
QEMU's device model runs → ioctl(KVM_RUN) again
(full round trip to userspace and back)When people say “VM exits are expensive,” they mean the second kind. The first is comparatively cheap — a mode switch and some state shuffling, but no userspace transition.
This distinction is also why the in-kernel GIC exists at all, and why the interrupt-related warning on Raspberry Pi from article 2 was worth understanding but not worrying about. It meant one interrupt fast path falls back to trapping — more exits than ideal, but cheap ones.
How QEMU learns what happened: the shared kvm_run struct
When KVM_RUN does return, QEMU needs to know why. Passing that back through the ioctl return value would be far too limited, so KVM uses shared memory.
When a vCPU is created, KVM gives QEMU a small region mapped into both QEMU’s address space and the kernel’s — a struct kvm_run. KVM writes the exit details into it before returning; QEMU reads them immediately after. No copying, no additional system calls.
┌─────────────────────────────────────────┐
│ kvm_run struct (shared QEMU ⇄ KVM) │
│ │
│ exit_reason = KVM_EXIT_MMIO │ ← KVM writes this
│ mmio.phys_addr = 0x09000000 │ on exit
│ mmio.data = [0x48] ('H') │
│ mmio.is_write = 1 │
│ mmio.len = 1 │
└─────────────────────────────────────────┘exit_reason is the dispatch key. KVM_EXIT_MMIO means the guest accessed a memory-mapped device region — the common case on ARM64, and exactly the mechanism from article 4. QEMU looks up which device model owns that address, calls it, and loops.
Follow the example concretely. The guest writes the character H to the UART at 0x09000000:
- The guest executes a store instruction. It has no idea anything unusual is happening.
- The address is in a device region, so the CPU traps. VM exit.
- KVM sees it’s MMIO to an address it doesn’t handle in-kernel. It fills in
kvm_runand returns fromioctl. - QEMU wakes, reads
exit_reason, dispatches to its PL011 UART model. - The UART model writes
Hto whatever the serial console is connected to — your terminal. - QEMU calls
ioctl(KVM_RUN)again. The guest resumes at the next instruction.
Every character of output from a guest’s serial console runs that loop. When you watch an OS installer scroll past over virsh console, you are watching this cycle execute thousands of times.
Why this is the foundation for everything that follows
With the round trip made concrete, several later topics stop being arbitrary.
VirtIO (article 6) is the architectural response to expensive exits. If each round trip costs, the strategy is to make each one carry more work. VirtIO batches many operations behind a single notification, amortizing one exit across dozens of requests. That argument only lands once you know what’s being amortized.
vCPU pinning matters because the thread carrying KVM_RUN is scheduled by ordinary Linux. Pin it to a core and guest state stays warm in that core’s caches across exits; let it migrate and every exit pays cold-cache costs.
QMP versus KVM_RUN are two channels people conflate. QMP is a socket libvirt uses to manage QEMU — hot-plug a disk, take a snapshot. KVM_RUN is the ioctl QEMU uses to run the guest. Different mechanisms, different purposes, no overlap.
The one-line version worth keeping: ioctl(vcpu_fd, KVM_RUN) is QEMU saying “go,” the guest then running natively on a real core until it trips on something, and the call returning only when there’s a device for QEMU to handle. That call, looped forever, is what running a VM physically is.
Summary
- A file descriptor is an integer indexing a per-process table of open kernel objects.
ioctlis the generic system call for device-specific commands thatreadandwritecan’t express.- KVM is driven entirely through
ioctlon/dev/kvm, via a three-level ladder: system fd → VM fd → vCPU fd. KVM_RUNon a vCPU fd enters the guest and blocks for as long as the guest runs natively — potentially millions of instructions.- Cheap exits are handled inside KVM (timers, the in-kernel GIC) and never wake QEMU. Expensive exits return to userspace so a QEMU device model can run.
- KVM reports why it exited through the shared
kvm_runstruct;exit_reasontells QEMU which device model to dispatch to. - Every character on a guest’s serial console is one full round trip through this loop.
Comments