Subtitle: Reducing the OS burden while taking advantage of new hardware features Xen is a hypervisor using a microkernel design that allows running multiple concurrent operating systems on the same hardware. One of the key features of Xen is that it is OS agnostic, meaning that any OS (with proper support) can be used as a host. Xen has a long history going back to the 90s when it was designed and the early 2000s when it was released. As a consequence of this, many of the assumptions and virtualization techniques backed into it are now superseeded by new hardware features, that make virtualization more transparent from an OS point of view. This talk provides an overview on the different kind of guests supported by Xen and how these new hardware features are used in order to improve and evolve them. It also describes the design and implementation of a new guest type, called PVHv2, and how it can be used as a control domain (Dom0). Also see: https://fosdem.org/2017/schedule/event/iaas_towahvm/

1. Towards a HVM-like Dom0 for Xen
Roger Pau Monn´e roger.pau@citrix.com
Brussels – February 4th, 2017

2. Xen Architecture Current Dom0 interfaces Designing a new interface Conclusions
Xen Architecture (type-1 hypervisor)
Hardware CPU MMU ...
Xen
syslog
xterm
xorg
...
Guest 1
Control
Domain
Linux, FreeBSD
or NetBSD
Guest 2
Brussels – February 4th, 2017 Towards a HVM-like Dom0 for Xen 2 / 18

3. Xen Architecture Current Dom0 interfaces Designing a new interface Conclusions
Type-2 hypervisor architecture
Hardware CPU MMU ...
syslog
xterm
xorg
...
Guest 1 Guest 2
Operating System
Brussels – February 4th, 2017 Towards a HVM-like Dom0 for Xen 3 / 18

4. Xen Architecture Current Dom0 interfaces Designing a new interface Conclusions
Current Dom0 interface
Due to the nature of the Xen architecture, a different interface
from the native one is used in order to perform several tasks:
MMU and privileged instructions.
CPU handling.
Setup and delivery of interrupts.
ACPI tables.
Brussels – February 4th, 2017 Towards a HVM-like Dom0 for Xen 4 / 18

5. Xen Architecture Current Dom0 interfaces Designing a new interface Conclusions
MMU and privileged instructions
Classic PV Dom0 uses the PV MMU:
Specific Xen MMU code in OSes.
Very intrusive.
Limited to 4KB pages.
Involves using hypercalls in order to setup page tables.
Hypercalls are used in order to request the hypervisor to
execute privileged instructions on behalf of the guest.
Brussels – February 4th, 2017 Towards a HVM-like Dom0 for Xen 5 / 18

6. Xen Architecture Current Dom0 interfaces Designing a new interface Conclusions
CPU handling
Native PV
Boot time enumeration ACPI MADT Hypercalls
AP bringup Local/x2 APIC Hypercalls
Hotplug ACPI GPE and processor objects1 Xenstore
1
Still under discussion
Brussels – February 4th, 2017 Towards a HVM-like Dom0 for Xen 6 / 18

7. Xen Architecture Current Dom0 interfaces Designing a new interface Conclusions
Setup and delivery of interrupts
On x86 systems interrupts are delivered from the APIC to the
CPU. There are several kinds of interrupts:
Legacy PCI: implemented using side-band signals, delivered to
the IO APIC and then injected into the local APIC
MSI/MSI-X: implemented using in-band signals delivered
directly to the local APIC.
Configuration of interrupts is done from the PCI configuration
space.
Brussels – February 4th, 2017 Towards a HVM-like Dom0 for Xen 7 / 18

8. Xen Architecture Current Dom0 interfaces Designing a new interface Conclusions
Setup and delivery of interrupts
Classic PV guests don’t have an emulated APIC.
Interrupts are delivered using event channels, the
paravirtualized interrupt interface provided by Xen.
Configuration of interrupts is performed using hypercalls.
Brussels – February 4th, 2017 Towards a HVM-like Dom0 for Xen 8 / 18

9. Xen Architecture Current Dom0 interfaces Designing a new interface Conclusions
Setup and delivery of interrupts
Hardware
CPU APIC MMU ...
Xen
Guest 1
Hardware
Domain
Event channel driver
Guest 2
Brussels – February 4th, 2017 Towards a HVM-like Dom0 for Xen 9 / 18

10. Xen Architecture Current Dom0 interfaces Designing a new interface Conclusions
ACPI tables
Two different kind of ACPI tables can be found as part of a
system description:
Static tables: binary structure in memory that can be directly
mapped into a C struct.
Dynamic tables: described using ACPI Machine Language
(AML), an AML parser is required in order to access them.
They can contain both data and methods.
On a classic PV Dom0 all tables are passed as-is to Dom0,
and that forces Xen to use side-band methods for CPU
enumeration.
Brussels – February 4th, 2017 Towards a HVM-like Dom0 for Xen 10 / 18

11. Xen Architecture Current Dom0 interfaces Designing a new interface Conclusions
ACPI tables
Xen can only parse information from static ACPI tables.
But there’s information required by Xen that resides in
dynamic tables:
Hotplug of physical CPUs.
CPU C states.
Sleep states.
Dom0 has to provide this information to Xen.
Although it would be possible for Xen to import a simple
AML parser, there can only be one OSPM, so Xen could only
look at the tables, but not execute any method.
Brussels – February 4th, 2017 Towards a HVM-like Dom0 for Xen 11 / 18

12. Xen Architecture Current Dom0 interfaces Designing a new interface Conclusions
A new interface for PVH Dom0
As close as possible to the native interface.
Only resort to hypercalls or similar options as last-resort.
Take advantage of the hardware virtualization extensions.
Brussels – February 4th, 2017 Towards a HVM-like Dom0 for Xen 12 / 18

13. Xen Architecture Current Dom0 interfaces Designing a new interface Conclusions
MMU
Use the hardware virtualization extensions in order to provide
a stage-2 page table for the guest:
Completely transparent from a guest point of view.
Guest can use the virtual MMU provided by the hardware.
Can use pages bigger than 4KB (2MB, 1GB).
No need for any modification of the OS.
Brussels – February 4th, 2017 Towards a HVM-like Dom0 for Xen 13 / 18

14. Xen Architecture Current Dom0 interfaces Designing a new interface Conclusions
Interrupt management
Provide Dom0 with an emulated local APIC and IO APICs.
Configuration of MSI/MSI-X interrupts from physical devices
using the PCI configuration space.
Brussels – February 4th, 2017 Towards a HVM-like Dom0 for Xen 14 / 18

15. Xen Architecture Current Dom0 interfaces Designing a new interface Conclusions
Interrupt management
Hardware
CPU APIC MMU ...
Xen
Guest 1
Hardware
Domain
vAPIC
Guest 2
Brussels – February 4th, 2017 Towards a HVM-like Dom0 for Xen 15 / 18

16. Xen Architecture Current Dom0 interfaces Designing a new interface Conclusions
ACPI tables
Provide Dom0 with the correct CPU topology in ACPI tables
(MADT).
Provide an extra SSDT table that contain processor objects
for the Dom0 vCPUs2.
Hide native processor objects from Dom0 using the STAO.
2
Still under discussion
Brussels – February 4th, 2017 Towards a HVM-like Dom0 for Xen 16 / 18

17. Xen Architecture Current Dom0 interfaces Designing a new interface Conclusions
Conclusions
Introduce a new interface, try to reduce Xen-specific code in
OSes.
Take advantage of hardware virtualization extensions.
Reduce the maintainership burden of OSes with Xen support.
Simplify the Dom0 interface, in order to promote Xen support
between OSes.
Brussels – February 4th, 2017 Towards a HVM-like Dom0 for Xen 17 / 18

18. Xen Architecture Current Dom0 interfaces Designing a new interface Conclusions
Q&A
Thanks
Questions?
Brussels – February 4th, 2017 Towards a HVM-like Dom0 for Xen 18 / 18