Why KVM Needs the CPU: ARM64 EL2, VHE, and the Virtualization Extension

The previous post said that KVM turns the running Linux kernel into a hypervisor. That is accurate, but incomplete, and the gap is where a lot of otherwise sound understanding comes apart.

A kernel cannot simply decide to run guest operating systems at the processor’s full speed. It needs the processor’s cooperation: a hardware feature that sits unused in the chip until software switches it on. This post is about that feature: the problem it solves, what it looks like on ARM64, and how it relates to KVM. The last part matters most, because “is virtualization hardware or software?” has an answer that is neither.

The problem: a guest kernel runs privileged instructions

Remember the illusion from the previous post: every guest operating system believes it owns the hardware. That belief has consequences. A kernel does not just think it is in charge; it runs instructions that only something in charge is allowed to run. It changes page tables, the structures that map memory. It writes to device registers. It switches interrupts on and off. It sets up the memory management unit.

Put that kernel in a VM, and it will try all of these, because that is what kernels do as they start. There are two obvious options, and both are bad.

Option one: let it run them on the real processor. The guest reconfigures the real memory management unit, writes to real device registers, and brings down the host and every other guest with it.

Option two: examine every instruction in software. Check each instruction before it runs, and when it is privileged, imitate its effect on pretend hardware instead. That works, and software-only virtualization does broadly this. But the cost is enormous: software is now involved in every instruction the guest runs, including the vast majority, which are harmless arithmetic and jumps.

Neither is necessary if the processor itself can tell the difference.

The fix is in the chip

Modern processors include a virtualization extension: a hardware feature that adds a privilege level above the normal kernel level, designed for a hypervisor to occupy.

With it, the guest kernel runs at its usual privilege level, fully believing it is in charge. Its ordinary instructions, the arithmetic, the jumps and the memory accesses, run directly on a real core at full speed, with no software involved. But the moment it tries something that would affect the real hardware, the processor traps: it stops the guest, and hands control up to the hypervisor.

That is what makes it affordable. The expensive interventions happen only for privileged operations, which are rare. Everything else runs at full speed, so the software no longer has to watch every instruction; it only handles the exceptions.

That trap is a VM exit: a transition, triggered by the hardware, from running the guest back to the hypervisor. This post covers why VM exits happen. What happens during one, how control reaches QEMU, what information crosses over, and why some exits cost far less than others, is the subject of post 1.5.

ARM64’s privilege levels, and the x86 equivalents

ARM calls its privilege levels Exception Levels, shortened to EL. Higher numbers mean more privilege, the opposite of x86’s numbering of rings.

Level	Who runs there
EL0	Applications, including a guest’s applications
EL1	Operating system kernels, including a guest’s kernel, where it has always expected to run
EL2	The hypervisor, which the guest cannot see or reach
EL3	Secure firmware, which has nothing to do with virtualization, and which KVM does not touch

On x86, the same idea is called VT-x on Intel processors and AMD-V on AMD ones, and the extra level is called “root mode” rather than given a number. The names differ, but the structure is the same: if you know x86 virtualization, EL2 is root mode.

VHE: why EL2 alone was not enough

ARM’s original design for virtualization assumed the hypervisor would be a small, purpose-built program living at EL2. EL2 was designed for that: a lean environment, with its own set of registers, deliberately unlike EL1.

Linux is not that. It is a large, general-purpose kernel, written to run at EL1, with KVM as part of it. Under the original design, running KVM meant splitting the kernel: most of Linux at EL1, and a small piece at EL2 handling the switches between host and guest. It worked, but every crossing between the two cost time, and the split was awkward.

VHE, the Virtualization Host Extensions, fixed this. Added in ARMv8.1, VHE lets a kernel written for EL1 run at EL2, by redirecting its accesses to EL1’s registers to EL2’s own. The host kernel, KVM included, can then run at EL2, with guests at EL1 below it. The awkward split is no longer needed.

So on processors with VHE, KVM runs in its efficient arrangement. On older ARM processors without it, KVM still works, using the split. The Raspberry Pi 5’s Cortex-A76 cores have VHE, and the kernel’s start-up messages confirm it.

Seeing it on real hardware

Here are the kernel’s KVM messages on my Raspberry Pi 5, captured on home1, the SD card used in Part 2:

home1@home1:~ $ dmesg 2>/dev/null | grep -i kvm
[    0.046684] kvm [1]: nv: 554 coarse grained trap handlers
[    0.046799] kvm [1]: IPA Size Limit: 40 bits
[    0.046811] kvm [1]: GICV region size/alignment is unsafe, using trapping (reduced performance)
[    0.046835] kvm [1]: vgic interrupt IRQ9
[    0.046846] kvm [1]: VHE mode initialized successfully

Every idea in this post appears in those five lines.

VHE mode initialized successfully confirms not just that KVM found a usable virtualization extension, but that it is running in the efficient arrangement described above: the host kernel at EL2, with no split. It is the moment the unused hardware feature was switched on, recorded as the kernel started.

IPA Size Limit: 40 bits is about the guest’s view of physical memory. When a guest uses what it believes is a physical address, it is really an intermediate physical address, or IPA, which the hardware then translates into a real address in the host’s memory. 40 bits allows addresses across 1 TiB. That is the size of the address space a guest can use, not memory it gets for free: every guest’s memory still has to come from the host’s real memory.

vgic interrupt IRQ9 is about interrupts. The GIC, or Generic Interrupt Controller, is the ARM hardware that delivers interrupts from devices to the processor cores. Guests need interrupts too, and several VMs cannot safely share the real controller, so KVM gives each VM a virtual one, the vGIC.

GICV region size/alignment is unsafe, using trapping (reduced performance) is specific to this hardware. Part of the interrupt controller cannot be mapped straight through to guests on this board, so KVM catches the guests’ accesses to it and handles them itself, inside the kernel. That means extra VM exits, but quick ones, handled without QEMU. It does not affect whether acceleration works.

nv: 554 coarse grained trap handlers is about nested virtualization: running a hypervisor inside a guest. It is not used in this series.

Is virtualization hardware or software?

Now for the question this post set out to answer. If virtualization is a processor feature, why is KVM part of the kernel? And if KVM is part of the kernel, what does the processor actually contribute?

They are two halves of one mechanism, and neither does anything alone.

The virtualization extension is a hardware capability that waits to be used. EL2 is in the Cortex-A76 from the moment it is made, built into the chip, and does nothing on its own. It is an engine with no driver.

KVM is the software that switches it on and uses it. When KVM starts, this is what happens:

So virtualization is a partnership with a strict dependency. The extension without KVM is an unused feature of the chip. KVM without the extension has nothing to do: it exists to use that hardware, and refuses to start on a processor that lacks it.

That has a useful consequence: /dev/kvm existing proves the whole chain worked. The file is only created when KVM has started successfully on a real virtualization extension. If it is there, the hardware feature is present, the software found it, and the door is open. That is why checking for it is the first step in post 2.1.

What happens without the extension

This is worth saying plainly, because it shows where the boundary is. Without a virtualization extension, you can still run VMs. They are just slow.

QEMU, short for Quick Emulator, is a complete machine emulator, and does not depend on KVM. It can imitate an entire computer in software, including a processor of a different kind, by translating the guest’s instructions, one block at a time, into instructions the host can run. That works without any hardware support, but it is far slower, because software is translating the guest’s instructions rather than letting them run on a real core.

Pair QEMU with KVM, and QEMU stops translating: the guest’s instructions run directly on the real cores, at close to full speed.

Combination	What it does	Result
QEMU alone	The whole machine in software	Works, slowly
QEMU with KVM	QEMU provides the devices, and KVM uses the extension for the processor and memory	Works, fast
KVM alone	Not possible: KVM only handles the processor and memory, and cannot be a disk or network card	Always needs a program to manage the VM

That program is usually QEMU, and it is QEMU throughout this series. Others exist, such as Firecracker, crosvm and kvmtool, which use KVM in the same way. The next post looks at how the work is divided.

The second line has a consequence specific to ARM64 hosts. KVM can only speed up guests whose instruction set matches the host’s. An ARM64 guest on an ARM64 host runs directly: the Cortex-A76 runs the guest’s instructions itself, so KVM applies, and it is fast. An x86 guest on the same host cannot: those are not ARM instructions, and no extension makes an ARM core run them. QEMU falls back to translating them.

That is not a limitation of KVM or of the Raspberry Pi. It is what hardware acceleration means, on any processor: the guest and host architectures must match, and when they do not, you are emulating, which is slow. Everything in this series that uses KVM uses ARM64 guests.

Summary

• A guest kernel runs privileged instructions. Letting them reach the real hardware is unsafe, and examining every instruction in software is slow.
• A virtualization extension adds a privilege level above the kernel. Guest code runs at full speed until it does something privileged, which traps to the hypervisor. That trap is a VM exit.
• On ARM64, the hypervisor level is EL2, above EL1 for kernels and EL0 for applications. On x86, the equivalent is VT-x or AMD-V root mode.
• VHE lets a whole Linux kernel run at EL2 without changes, which is the efficient arrangement KVM uses on modern ARM processors. dmesg reports it.
• The IPA size is the size of a guest’s physical address space, not extra memory.
• The hardware extension and KVM work together: the extension is hardware waiting to be used, and KVM switches it on. KVM refuses to start without it.
• /dev/kvm existing proves the whole chain worked.
• KVM needs a program to manage each VM, usually QEMU.
• KVM only speeds up guests with the same architecture as the host. x86 guests on ARM64 are emulated, slowly.