In this talk, StorPool will present the concept of nested virtualization and PCI pass-through. Nested virtualization is when you have virtual machines running inside other virtual machines. If done right, nested virtualization is extremely suitable for testing large virtualized environments, without the need for having hundreds of physical servers.
This approach empowers system administrators by reducing the time and effort needed to run/test multiple dynamic environments. This is done by having easier automation/provisioning, API control and much more. In a nested virtualization solution, the physical hardware is sliced and diced fast and easy, which gives you a large set of opportunities to experiment without the need for huge investments in physical hardware or time and effort needed to reinstall/set-up software.
Join this talk to learn more about why and how to do nested virtualization with OpenNebula and achieve an inception moment of having virtual machines, inside virtual machines, inside virtual machines…
7. VMCS Shadowing
● Hardware feature
● Accelerates nested virtualization
● Available in most CPUs since 2013 (Haswell)
Source: https://software.intel.com/en-us/blogs/2014/12/12/enabling-virtual-machine-control-structure-shadowing-on-a-nested-virtual-machine
8. PCI Passthrough
Allows guests to have direct
exclusive access to PCI devices
● I/O MMU virtualization (IOMMU)
VT-d
AMD-Vi
Mostly used for GPU and NIC
Host
GPU1
Hypervisor / VMM
VM
Host
driver
driver
VM
driver
VM
driver
GPU2 GPU3GPU0
PCIe
IOMMU / VT-d
9. SR-IOV
● Single Root I/O Virtualization
● One physical device appears as
multiple virtual functions (VF)
● Allows different VMs to share a
single PCIe hardware
● Mostly used for NIC
Host
NIC
VF1
Hypervisor / VMM
VM
Host
driver
driver
VM
driver
VM
driver
VF2 VF3PF
PCIe
IOMMU / VT-d
10. ACS and IOMMU Groups
Host
GPU1
Hypervisor / VMM
VM
Host
driver
driver
VMVM
driver
VM
driver
GPU2 GPU3GPU0
IOMMU / VT-dRAM
ACS
PCIe
12. Host
Hardware Virtualization
Check it is available with: lscpu | grep vmx
Nested Virtualization
kvm-intel.conf: options kvm-intel nested=1
Check it in the guest with: lscpu | grep vmx
VMCS Shadowing
kvm-intel.conf: options kvm-intel enable_shadow_vmcs=1
13. PCI Passthrough & SR-IOV
PCI Passthrough
Enabled via kernel options:
Intel:
intel_iommu=on iommu=pt
AMD:
amd_iommu=pt
SR-IOV
● Ensure SR-IOV and VT-d are enabled in BIOS.
● Setup VFs:
# echo '8' > /sys/class/net/eth3/device/sriov_numvfs
14. ACS and IOMMU Groups
● Check ACS is supported
lspci -vv | egrep “Access Control Services”
https://heiko-sieger.info/iommu-groups
-what-you-need-to-consider/
● Check IOMMU groups
for a in /sys/kernel/iommu_groups/*; do find $a -type l;
done | sort --version-sort