How QEMU Emulates Hardware: Device Models, MMIO, and TCG

The previous post said that QEMU imitates the devices KVM deliberately does not.

“QEMU emulates devices” is often treated as a kind of magic: an ordinary program somehow produces a virtual disk or network card from nothing, and the guest operating system is completely fooled.

There is no magic. The whole illusion rests on one mechanism, applied the same way to every piece of virtual hardware. Once you see it, the performance of virtualization makes sense: why virtio exists, and why some operations are so much slower in a VM than on real hardware.

First: QEMU does two different jobs

One point of confusion to clear up first. QEMU does two entirely different jobs, depending on which program you run:

	Full-system emulation	User-mode emulation
Program	qemu-system-aarch64, and others named qemu-system-*	qemu-aarch64, and others named qemu-*
What it does	Imitates a whole computer: processor, memory, disks, network cards and firmware, and starts an operating system	Runs one program built for another processor, translating its requests to the host’s kernel
A virtual machine?	Yes	No: there is no operating system and no machine

Full-system emulation builds a complete virtual computer, and starts a whole guest operating system on it. It is what libvirt starts for every VM, and it is what this series is about.

User-mode emulation runs a single program built for a different processor, directly on the host, by translating its calls to the kernel as they happen. It does not build a machine or start an operating system. It is how you run an x86 program on an ARM computer without a VM, and it is what Docker uses, through a kernel feature called binfmt_misc, to run containers built for another processor. Useful, but not virtualization.

So if a command starts with qemu-system-, it is building an entire computer.

The trick: devices live at memory addresses

Emulation rests on a fact about real hardware.

When an operating system wants to send a character to a serial port, it does not call a function. It writes to a particular memory address. That address is connected, by the motherboard’s wiring, to the serial port’s chip rather than to memory, so the write goes to a register on the chip instead.

This is called memory-mapped I/O, or MMIO, and it is how the processor talks to almost every device on a modern ARM system. Devices are not special things with a private channel to the processor. They are regions of the address space.

So the processor’s whole interface to a device comes down to this: writing to certain addresses, and reading from certain addresses. Nothing more. And that is an interface software can easily imitate.

QEMU lays out the guest’s address space with some regions marked as devices rather than memory. When the guest reads or writes in one of those regions, the access is not carried out as a memory access. It traps, and QEMU runs a function in its own code.

That function is the device. A virtual serial port is a few hundred lines of C that print a character to your terminal when the guest writes one. A virtual disk controller is C that receives the guest’s read request, and fetches the bytes from a qcow2 file on the host.

What the code looks like

Here is an illustration of the shape of that code. It is not QEMU’s real serial port model: it is a made-up, simplified serial port, with invented register positions, to show that there is nothing more to it than this. QEMU’s real model of the ARM PL011 serial port is in hw/char/pl011.c, and is longer, but built the same way.

First, QEMU connects a region of the guest’s address space to two functions, using a structure called MemoryRegionOps, whose field names are real:

/* tells QEMU which functions to run when the guest touches this region */
static const MemoryRegionOps example_uart_ops = {
    .read  = example_uart_read,     /* run when the guest reads here */
    .write = example_uart_write,    /* run when the guest writes here */
    .endianness = DEVICE_NATIVE_ENDIAN,
    .valid.min_access_size = 1,
    .valid.max_access_size = 4,
};

Then the function itself. When the guest writes to one of the device’s addresses, the processor traps, and QEMU calls this with the position inside the region, offset, and the value written:

/* an illustration, not QEMU's real PL011 model */
static void example_uart_write(void *opaque, hwaddr offset,
                               uint64_t value, unsigned size)
{
    ExampleUARTState *s = opaque;

    switch (offset) {
    case 0x00:   /* data register: the guest is sending a character */
        putchar((char)value);    /* no wires: print it on the host */
        fflush(stdout);
        s->status |= TX_EMPTY;   /* report "ready for the next one" */
        break;

    case 0x04:   /* interrupt settings */
        s->irq_mask = value;     /* just remember them in a variable */
        break;

    default:     /* a register this device does not have */
        log_bad_access(offset);
        break;
    }
}

There is no magic in it. Every device a guest sees is a piece of software like this: code imitating a chip, by responding to the same reads and writes the real chip would. The guest’s driver cannot tell the difference, because from its point of view there is none. It writes to an address, and the right thing happens.

How the guest knows where the devices are

That raises the obvious question: how does the guest know there is a serial port at 0x09000000 in the first place?

On ARM, nothing is discovered by guesswork: the machine has to be described to the operating system, in a table of what hardware exists and at which addresses. There are two kinds of table:

Table	Where it is used
A device tree	Small boards and embedded systems, and VMs whose kernel is started directly, without firmware
ACPI tables	ARM servers, and VMs that start through UEFI firmware

QEMU generates both for its virt machine, describing the hardware it imitates. A guest that starts through UEFI firmware, as lab1 and the other VMs in this series do, is usually given the ACPI tables, which is why lab1’s XML includes <acpi/>, as post 3.4 shows. Either way, the guest’s kernel reads the table, finds an entry such as “a PL011 serial port at 0x09000000”, loads its ordinary driver for real PL011 hardware, and starts writing to that address. The driver has no idea it is talking to C code.

The PL011 is ARM’s serial port design. A Raspberry Pi has real PL011 serial ports, and Linux calls them ttyAMA0 and so on, the same name it uses for QEMU’s imitation of one, as post 3.6 explains.

A device tree, simplified

A device tree is the easier of the two to read. Here is a shortened example of the kind QEMU generates for its virt machine, for a guest with two Cortex-A57 processors and GIC version 3. A VM like lab1 would show the Pi’s own processor and GIC version 2 instead. Again, it is only here to show that there is no magic:

/dts-v1/;

/ {
    model = "linux,dummy-virt";              /* QEMU's artificial 'virt' board */
    compatible = "linux,dummy-virt";
    #address-cells = <0x02>;
    #size-cells = <0x02>;

    cpus {
        #address-cells = <0x01>;
        #size-cells = <0x00>;
        cpu@0 { device_type = "cpu"; compatible = "arm,cortex-a57"; reg = <0x00>; };
        cpu@1 { device_type = "cpu"; compatible = "arm,cortex-a57"; reg = <0x01>; };
    };

    memory@40000000 {                         /* memory starts at 0x40000000 */
        device_type = "memory";
        reg = <0x00 0x40000000 0x00 0x80000000>;   /* 2 GiB */
    };

    intc@8000000 {                            /* the interrupt controller */
        compatible = "arm,gic-v3";
        interrupt-controller;
        reg = <0x00 0x08000000 0x00 0x010000>,
              <0x00 0x080a0000 0x00 0xf60000>;
    };

    pl011@9000000 {                           /* the serial port */
        compatible = "arm,pl011", "arm,primecell";   /* which driver to use */
        reg = <0x00 0x09000000 0x00 0x1000>;          /* where it is, and its size */
        interrupts = <0x00 0x01 0x04>;                /* which interrupt it raises */
    };

    /* ... the PCI bus, virtio devices and flash memory follow ... */
};

The pl011@9000000 entry at the end is the serial port, at 0x09000000, exactly where the guest’s driver will write.

This is also why the virt machine matters on ARM64. It is not an imitation of any real board: it is an artificial machine, defined by QEMU and designed for VMs. On x86, the usual machine types, pc and q35, imitate real historical chipsets, oddities included. ARM64 never had to carry that history.

QEMU’s two halves

By now, QEMU’s job with devices should be clear. But it also has a separate problem: running the guest’s processor instructions.

The devices never change. As the previous post explained, QEMU still works without KVM, only slowly, and either way, the devices are imitated by QEMU, as an ordinary program. KVM has no devices of its own, and never gains any. There is one small exception: KVM implements a few parts that need to be very fast, such as the interrupt controller, inside the kernel. That is an optimization, and the general rule stands.

The processor has two possible engines:

• TCG, the Tiny Code Generator, is QEMU’s software processor. It reads blocks of the guest’s instructions, translates them into equivalent instructions for the host, keeps the results, and runs them, translating as it goes. This is true software emulation, and it is the only option when the guest’s and host’s architectures differ. An x86 guest on an ARM64 host must use TCG, because no hardware feature makes an ARM core run x86 instructions.
• KVM is the hardware path from post 1.2. Instead of translating, QEMU asks KVM to run the guest’s code directly on a real core, using the virtualization extension. KVM hands control back to QEMU only when the guest touches a device.

In libvirt’s XML, the choice is <domain type='kvm'> or <domain type='qemu'>, and it is the biggest single factor in a VM’s performance.

What a device access costs

Put the two halves together, and the central fact of virtualization performance falls into place.

With KVM, guest code runs at full speed. Arithmetic, loops and memory accesses happen directly on the real processor, with no involvement from KVM or QEMU: millions of instructions in a row can run without either of them waking up.

But a device access cannot complete on the processor, because there is no real device. It has to trap out to QEMU’s code.

So the cost model of a virtual machine is: full speed for computation, and a fairly expensive round trip for every device access.

That one imbalance explains a great deal. It is why work that mostly computes runs in a VM at close to its full speed, while work that mostly reads and writes falls behind. It is why a virtual serial console is fine for a login prompt: people type slowly, so one trap per character goes unnoticed. And it is why the same approach would be disastrous for a network card handling a hundred thousand packets a second.

It is also why virtio exists. If each device access is expensive, the answer is to make each one do more work: in other words, to batch. Post 1.6 covers that. To follow the argument, though, it helps to know what the round trip involves: what “control leaves the guest” means in practice, what information crosses over, and why some traps cost far less than others. That is the next post.

Summary

• qemu-system-* imitates a whole machine and starts an operating system. qemu-* runs a single program built for another processor. Only the first is virtualization.
• On real hardware, devices are reached through memory-mapped I/O: the processor writes to addresses connected to chips rather than memory.
• QEMU uses this by marking regions of the guest’s address space as devices. An access there traps, and runs a function in QEMU. That function is the device.
• The guest finds out what exists, and where, from a table QEMU provides: a device tree, or, for guests that start through UEFI like those in this series, usually ACPI tables. The guest uses its ordinary drivers.
• QEMU has two halves. The devices are always imitated by QEMU. The processor is either translated in software, by TCG, or run directly, by KVM.
• With KVM, computation runs at full speed, but every device access costs a trap out to QEMU. That imbalance underlies every performance decision in virtualization.