1.4 · How the stack works
How QEMU Emulates Hardware: Device Models, MMIO, and TCG
The previous article established that QEMU emulates the devices KVM deliberately doesn’t.
“QEMU emulates devices” is a sentence that gets repeated constantly and explained rarely. It sounds like a black box: somehow, a program conjures a disk out of nothing and the guest is fooled.
There’s no black box. The mechanism is one trick, applied consistently, and once you can see it every performance characteristic in the rest of this series follows from it — including why VirtIO exists, why some operations are catastrophically slower in a VM than on bare metal, and what a VM exit actually costs.
First: QEMU is two different programs
Worth clearing up before anything else, because the name is overloaded and the confusion is common.
FULL-SYSTEM emulation USER-MODE emulation
(what this series is about) (a different job entirely)
emulates a WHOLE computer: runs ONE foreign binary
CPU, RAM, disk, NIC, firmware, by translating its syscalls
the works — boots an OS (no OS, no virtual machine)
qemu-system-aarch64 qemu-aarch64Full-system emulation boots a complete guest operating system on a virtual machine. That’s qemu-system-aarch64, and it’s what libvirt launches for every VM.
User-mode emulation runs a single foreign-architecture binary on your existing OS, translating its system calls to the host kernel. It’s how you run an x86 binary on an ARM machine without a VM, and it’s what Docker’s binfmt_misc cross-architecture support uses under the hood. Useful, but not virtualization.
Everything below concerns qemu-system-*. If a command starts with qemu-system-, it’s building a machine.
The trick: devices live at memory addresses
Here’s the fact about real hardware that makes emulation possible.
When an operating system wants to send a byte to a serial port, it doesn’t call a function. It writes to a specific memory address — one that the hardware has physically wired to the serial chip. The CPU issues what looks like an ordinary memory write, but no RAM is involved. The address decoder on the board routes it to the UART instead.
This is memory-mapped I/O, or MMIO, and it’s how the CPU talks to essentially every device on a modern ARM system. Devices are not special entities the CPU has a private channel to. They’re regions of the address space.
Which means the CPU’s entire interface to a device is: writes to certain addresses, and reads from certain addresses. Nothing else. And that’s an interface software can impersonate.
On real hardware:
┌──────┐ writes byte to ┌─────────────┐
│ CPU │ ──address 0x09000000──► │ UART (real │ ──► out the
│ │ │ serial chip)│ serial port
└──────┘ └─────────────┘
Under QEMU:
┌──────────┐ guest writes to ┌──────────────────┐
│ guest │ ─0x09000000────────► │ QEMU notices the │ ──► writes to
│ CPU │ (a trap!) │ write, RUNS C │ your terminal
│ │ │ CODE that mimics │
└──────────┘ │ a UART │
└──────────────────┘QEMU constructs a guest address space in which certain regions are marked as device regions rather than RAM. When the guest accesses one, the access doesn’t complete as a memory operation — it traps, and QEMU runs a function.
That function is the device. A virtual UART is a few hundred lines of C that, when written to, puts a character on your terminal. A virtual disk controller is C code that, when the guest issues a read, returns bytes from a qcow2 file on the host. A virtual network card is C code that, when the guest hands it a packet, passes it to the host’s network stack.
There is no magic anywhere in this. Every device your guest sees is a software model — a chunk of code pretending to be a chip by responding to the same address accesses the real chip would respond to. The guest’s driver cannot tell the difference, because from the driver’s perspective there is no difference: it writes to an address and something happens.
0x09000000 in the diagram is not arbitrary, incidentally. It’s the address of the first UART on QEMU’s ARM virt machine — the board type every guest in this series runs on.
Why the guest’s firmware finds these devices at all
One question this raises: how does the guest know a UART is at 0x09000000?
On a physical ARM board, the answer is a device tree — a data structure describing what hardware exists and where, passed to the kernel at boot. ARM has no equivalent of x86’s legacy enumeration conventions, so the hardware layout has to be described explicitly.
QEMU generates a device tree describing the machine it’s emulating, and hands it to the guest at boot. The guest kernel reads it, sees “there’s a PL011 UART at 0x09000000,” loads the PL011 driver, and starts writing to that address. The driver is the standard, unmodified Linux driver for real PL011 hardware. It has no idea it’s talking to C code.
This is also why the virt machine type matters on ARM64. It’s not emulating any real-world board — it’s a synthetic machine defined by QEMU, with a clean layout and no legacy hardware. On x86, machine types like pc and q35 emulate actual historical chipsets, complete with their quirks. ARM64 skipped that inheritance.
QEMU’s two halves
So QEMU has a device job. It also has a CPU job — and how it handles the second is exactly where KVM enters.
┌─────────────────────────────────────────────┐
│ qemu-system-aarch64 │
│ │
│ HALF 1: the guest CPU │
│ ┌────────────────────────────────────────┐ │
│ │ "how do I execute guest instructions?" │ │
│ │ │ │
│ │ option A: TCG — translate in software │ │ ← slow, no KVM
│ │ option B: KVM — run on real core │ │ ← fast, needs ext
│ └────────────────────────────────────────┘ │
│ │
│ HALF 2: the devices │
│ ┌────────────────────────────────────────┐ │
│ │ C models of: UART, disk, NIC, RTC, │ │
│ │ interrupt controller, PCI, USB… │ │ ← ALWAYS QEMU's job,
│ │ this is ALWAYS done in QEMU software │ │ KVM never touches it
│ └────────────────────────────────────────┘ │
└─────────────────────────────────────────────┘Half 2 never changes. Whether or not KVM is involved, devices are emulated by QEMU in userspace. KVM has no device models and never acquires any. (There’s a partial exception — KVM emulates a few latency-critical devices like the interrupt controller in-kernel for speed — but the general rule holds, and the exception is itself a performance optimization worth knowing about.)
Half 1 is the part with two implementations.
TCG — the Tiny Code Generator — is QEMU’s software CPU. It reads blocks of guest instructions, translates them into equivalent host instructions, caches the result, and executes the translation. It’s a just-in-time compiler from one instruction set to another. This is genuine emulation, and it’s the only option when guest and host architectures differ — an x86 guest on an ARM64 host has no alternative, because no amount of hardware assistance makes an ARM core execute x86 instructions.
KVM is the hardware path from article 2. QEMU asks KVM to run the guest, KVM executes guest code directly on a real core via the virtualization extension, and control returns to QEMU only on a trap.
The choice between them is what <domain type='kvm'> versus <domain type='qemu'> selects in libvirt’s XML, and it’s the single biggest performance factor in the entire stack.
The consequence: what a device access costs
Now combine the two halves, and the central performance fact of virtualization falls out.
Under KVM, guest code runs natively at full speed. Arithmetic, branches, memory accesses to actual RAM — all native, with no involvement from KVM or QEMU. Millions of instructions can execute without either one waking up.
But a device access can’t complete natively, because there’s no device. It has to trap out to QEMU’s C code.
guest runs on real core (KVM) ──┐
│ │ most instructions:
│ guest writes to a │ stay down here, fast
│ device address │
▼ │
TRAP — control leaves the guest ─┘
│
▼
QEMU's device model handles it
│ (writes to a file, sends a packet, prints a character)
▼
back into the guest, running natively againSo the cost model of a virtual machine is: native speed for computation, and a relatively expensive round trip for every device interaction.
That single asymmetry explains an enormous amount. It’s why CPU-bound workloads in VMs run at near-bare-metal speed while I/O-heavy ones don’t. It’s why a virtual serial console is fine for a login shell — humans type slowly, so a trap per character is irrelevant — but would be catastrophic for a network card processing a hundred thousand packets a second.
And it’s the reason VirtIO exists. If device access is the expensive operation, the winning strategy is to make each one carry more work — to batch. That’s article 6.
But the batching argument only lands if you know what the round trip actually involves: what “control leaves the guest” means mechanically, what data crosses the boundary, and why some traps are far cheaper than others. That’s the next article, and it’s the one that makes the rest of the series’ performance discussion concrete.
Summary
qemu-system-*emulates a whole machine and boots an OS.qemu-<arch>runs a single foreign binary. Only the first is virtualization.- Devices on real hardware are reached through memory-mapped I/O — the CPU writes to addresses that are wired to chips rather than RAM.
- QEMU exploits this by marking regions of the guest address space as device regions. An access there traps and runs a C function. That function is the device.
- On ARM64, the guest discovers what exists and where via a device tree QEMU generates. Guest drivers are the standard, unmodified Linux drivers.
- QEMU has two halves. Devices are always emulated in QEMU userspace. The CPU is either translated in software (TCG) or run natively on real hardware (KVM).
- Under KVM, computation runs at native speed but every device access costs a trap out to QEMU. That asymmetry is the foundation of every performance decision in virtualization.
Comments