XPDS16: Live Migration of vGPU - Xiao Zheng, Intel Asia-Pacific Research & Development Ltd.
•
1 like•961 views
GPU virtualization is hot in cloud usages including VDI, media processing, etc. While Intel GVT-g (a.k.a XenGT) helps unleash those compelling usages on Intel Processor Graphics, new requirements are emerging such as VM live migration with vGPU. In this session we will introduce the challenges of supporting vGPU live migration on current migration framework, then elaborate techniques to bring vGPU live migration into XenGT.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to XPDS16: Live Migration of vGPU - Xiao Zheng, Intel Asia-Pacific Research & Development Ltd.
Similar to XPDS16: Live Migration of vGPU - Xiao Zheng, Intel Asia-Pacific Research & Development Ltd. (20)
More from The Linux Foundation
More from The Linux Foundation (20)
Recently uploaded
Recently uploaded (20)
XPDS16: Live Migration of vGPU - Xiao Zheng, Intel Asia-Pacific Research & Development Ltd.
- 1. Live Migration of vGPU Aug 2016 Xiao Zheng xiao.zheng@Intel.com Kevin Tian kevin.tian@Intel.com
- 2. 2 Agenda • GPU Virtualization and vGPU Live Migration • vGPU Resources • Design and Solution • Current Status • Summary
- 3. 3 GPU Virtualization Usage Cases IT 2D/3D Office Productivity Desktop CADs Media Could Media Process 3D/Media Acceleration … Network Remote Framebuffer Streaming VDI Desktop#N
- 4. 4 XENGT Architecture – Mediated Pass-through XEN i915 GPU Qemu VGA Host Linux VM1 VM2 GFX Driver MPT Services vGT vGPU vGPU vPCI layout I/O Hooks Driver Hooks Address space balloon Address space balloon • pass-through for performance critical resource • Trap and emulate for privileged resource • Time-shared among VMs
- 5. 5 vGPU Live Migration Live Migration: Load balance, Maintenance, Fault recovery Unfortunately most of vGPU solutions do not support migration except GVT-g HW GPU GPU GPU GPU Hypervisor Guest vGPU Typically a GPU pass-through solution SRIOV HW GPU PF VF … VF Hypervisor Guest vGPU Typically a GPU SRIOV solution GVT-g with mediated pass-through XEN i915 GPU Qemu V G A Host Linux VM1 VM2 GFX Driver MPT Services vGT vGP U vGP U vPCI layo ut I/O Hook s Drive r Hook s Addre ss space balloo n Address space balloon GVT-g architecture (Mediation) make it possible for seamless live migration
- 6. 6 Live Migration of vGPU in GVT-g Highlight feature: • GVT-g is Open Source project, upstream ongoing • vGPU Live Migration follows existing hypervisor migration flow • 3D/2D/Media graphics workload seamless migrated between Servers or Local machine • Support Linux/Windows Guest • Live Migration Service downtime latency < 0.3 sec (Guest RAM 2GB, assigned 512MB vGPU memory, 10Gpbs adapter) VM1 Office uage VM2 CADs VM3* Media Process … VDI VM4 Media Process VM3 vGPU Server1 vGPU Server2 How?
- 7. 7 Demo: vGPU Live Migration with 3D workload https://www.youtube.com/watch?v=y2SkU5JODIY
- 8. 8 Agenda • GPU Virtualization and vGPU Live Migration • vGPU Resources • Design and Solution • Current Status • Summary
- 9. 9 Inside of vGPU instance Graphics Memory Render Engine Registers GPU Interrupt pass-through for performance critical resource Trap and emulate for privileged resource Time-shared among VMs Graphics Memory Render Engine Registers vGPU Interrupt XenGT Graphics Memory Render Engine Registers vGPU Interrupt GTT page table
- 10. 10 Challenge of Migrating vGPU Instance • When and how to migrate Graphics Memory • When and how to migrate Guest Graphics Page Table • When and how to migrate Render Engine State Pre-copy GPU dirty page logging Stop and Copy • Remove vGPU from scheduler • Save and migrate all vGPU state Resume/Post-Copy Restore vGPU state Add vGPU into scheduler Total migration time Service downtime
- 11. 11 Migration Policies for Different vGPU Resources Context: Render Engine GTT page table Graphics Memory Registers Copy and Restore Recreate Shadowing Track Dirty and Copy Recreate Shadowing
- 12. 12 Agenda • GPU Virtualization and vGPU Live Migration • vGPU Resources • Design and Solution • Current Status • Summary
- 13. 13 Guest GTT Page Table Migration VM0 VM1*VM0 HW GTT view Target HW GGTT view GTT 0 ~ 512MB Guest VM GTT view GTT 0 ~ 512MB VM2 VM2VM1 GTT = GGTT or PPGTT GMA = Graphics Mem Address GMA rebasing GM address = 0xA GM address = 0xB • Both GGTT and PPGTT are shadowed for Guest • GGTT required rebasing due to GGTT partition among VMs • Migration process actually: A. Copy entire Guest GTT page table B. Re-create the shadow page table for Guest on Target side C. Rebasing GGTT for GPU commands Migration GMA rebasing VM1 Graphics Memory Address rebasing: All vGPU cmds from Guest need to be rebased on new address in GVT-g before send to real GPU HW GMA rebasing
- 14. 14 Guest Graphics Memory Migration • Pre-copy: Logging dirty graphics memory pages • Stop-and-Copy: Migrate contents to target • Resume/Post-copy: Recreate GTT page table based on target mfn Problem: Intel® GPU page table entities has no Dirty or Accessed flags to track dirty pages Solution: Copy all used graphics memory to target. GTT 0 ~ 512MB Guest VM GTT view Source host Physical Mem Target host Physical Mem Source mfn Target mfn VM1 Migrate page content
- 15. 15 Render Engine State Migration Server1 Render Engine Render Context4 Render Context3 Render Context2 HW in idle Context1 completed Context0 completed GGTT address vGPU Guest submitted CTX Context in queue Migration happens at this point CTX N Render Context required to be migrated o Intel® GPU HW is context based o CTX locates in GGTT memory o Render engine state is contained within CTX Server2 Render Engine HW in idle
- 16. 16 Agenda • GPU Virtualization and vGPU Live Migration • vGPU Resources • Design and Solution • Current Status • Summary
- 17. 17 Current Status • Experimental support both KVMGT and XENGT • Platforms: Intel® 5th /6th Generation Intel® Core™ Processors • Benchmarks covered: Windows guest: Heaven, 3Dmark06, Trophic, Media encoding/decoding, Linux guest: lightsmark, 2D • Quality: 12hours overnight testing, migrating every 30sec • Timing: (Guest RAM 2GB including 512MB Graphics memory, 10Gbps adapter) Service downtime ~0.3sec Total migration time: ~1.7sec Pre-copy GPU dirty page logging Stop and Copy • Remove vGPU scheduler • Save and migrate all vGPU state Resume/Post-Copy Restore vGPU state Add vGPU into scheduler Total migration time ~1.7sec Service downtime ~0.3sec
- 18. 18 Summary • Need 3D/2D/Media workload in virtualization? GVT-g is the choice • Need GPU virtualization with migration support? GVT-g is the choice
- 19. 19 Resource Links • Project webpage and release: https://01.org/igvt-g • Project public papers and document: https://01.org/group/2230/documentation-list • Intel® IDF: GVT-g in Media Cloud: https://01.org/sites/default/files/documentation/sz15_sfts002_100_engf.pdf • XenGT introduction in summit in 2015: http://events.linuxfoundation.org/sites/events/files/slides/XenGT- Xen%20Summit-REWRITE%203RD%20v4.pdf • XenGT introduction in summit in 2014: http://events.linuxfoundation.org/sites/events/files/slides/XenGT- LinuxCollaborationSummit-final_1.pdf
- 20. 20 Notices and Disclaimers INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL® PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL’S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL® PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. INTEL PRODUCTS ARE NOT INTENDED FOR USE IN MEDICAL, LIFE SAVING, OR LIFE SUSTAINING APPLICATIONS. Intel may make changes to specifications and product descriptions at any time, without notice. All products, dates, and figures specified are preliminary based on current expectations, and are subject to change without notice. Intel, processors, chipsets, and desktop boards may contain design defects or errors known as errata, which may cause the product to deviate from published specifications. Current characterized errata are available on request. No computer system can provide absolute security under all conditions. Intel® Trusted Execution Technology (Intel® TXT) requires a computer with Intel® Virtualization Technology, an Intel TXT-enabled processor, chipset, BIOS, Authenticated Code Modules and an Intel TXT-compatible measured launched environment (MLE). Intel TXT also requires the system to contain a TPM v1.s. For more information, visit http://www.intel.com/technology/security Intel, Intel logo, Xeon, and Xeon Inside are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. *Other names and brands may be claimed as the property of others. Copyright © 2016 Intel Corporation. All rights reserved.