Presented by Mark Kilgard on July 27, 2016 at NVIDIA's Booth at SIGGRAPH.in Anaheim

1. Mark Kilgard, July 27, 2016
Virtual Reality Features of
NVIDIA GPUs

2. 2
Mark Kilgard
Principal System Software Engineer
OpenGL driver implementation & API evolution
Other interests
Programmable shading languages, original Cg GPU shading language
Web, vector graphics & path rendering, GPU rendering for 2D graphics
Also known for creating OpenGL Utility Toolkit (GLUT)
My Background

3. 3
GPU Challenges for Virtual Reality Rendering
1. Must render unique view for each eye
Two scene renders per frame
2. Wide field-of-view rendered
More pixels, distributed towards periphery
3. HMD lens magnifies rendered scene
Non-linear inverse lens warp
Allows the HMD lens magnification to provide
convincing sense of presence

4. 4
Magnification Effect of HMD Lens
Graph paper viewed & magnified through HMD lens

5. 5
Head Mounted Display Lens Effect
Barrel Distorted Images
on HMD Screen
Magnified Overlapping Wide Field-of-view
Binocular View

6. 6
Focus on Virtual Reality Features of Pascal GPUs
Virtual Reality rendering features in Pascal GPUs
GeForce 1080/1070/1060
NEW Quadro P6000 & P5000
As exposed in standard 3D APIs

7. 7
Background: NVIDA GPU Architecture Road Map
Our interest NVIDIA GPU architectures of interest: Pascal
What is Pascal mentioned on last slide?

8. 8
Conventional 3D Single-eye Scene

9. 9
Conventional 3D Single-eye Scene

10. 10
Wide Stereo Views of Same Scene
Left Right

11. 11
Left Right

12. 12
Right Left

13. 13
Left & Right Views Are
Similar but Different
Left and Right eye views have different occlusion
While left & right eye views look similar, must be draw independently
Really only different in a small translation in X
NVIDIA Single Pass Stereo supports generating left & right eye positions
GPU automatically makes left & right eye versions of each primitive
Then renders each view to a different framebuffer slice
Exposed in OpenGL by NV_stereo_view_rendering extension
Exposed for Direct3D 11 by NVAPI SinglePassStereoMode feature

14. 14
Image Difference of Two Views
− + 0.5 =
Left eye view Right eye view
Clamped difference image

15. 15
Left & Right Views Rendered Efficiently in One Pass

16. 16
Lens Matched Shading
Left Right

17. 17
Visualizing Lens Matched Shading

18. 18
Lens Matched Shading
Left Right

19. 19
Visualization of Lens Matched Shading Rendering

20. 20
Still Needs Non-linear Warp
Lens Matched Shading
rendering
Warped image suitable
for HMD display
Non-linear
image

21. 21
Still Needs Non-linear Warp
Lens Matched Shading
rendering
Warped image suitable
for HMD display
Non-linear
image
BUT substantially
less pixel motion
and distortion
than conventional
VR re-warping

22. 22
Example HMD Post-rendering Warp

23. 23
Lens Matched Shading
with Window Rectangle Testing

24. 24
Lens Matched Shading
with Window Rectangle Testing
Nothing in black corners is shaded or even rasterized
Yellow lines show overlaid 8 inclusive window rectangles
Same 8 window rectangles “shared” by each view’s texture array layer

25. 25
Lens Matched Shading
with Window Rectangle Testing

26. 26
What Didn’t Have to be Rendered
17% of left view pixels 17% of right view pixels

27. 27
Window Rectangle Testing
During Rasterization
New multi-vendor OpenGL extension
EXT_window_rectangles
Provides a set of window-space
rectangles, 8 for NVIDIA GPUs
Rasterizer either includes or excludes
rasterization from rectangles
Discards at speed-of-light
As if free to rasterize to irregular
rectangular regions
Window
Rectangles
Test
NEW
stage
OpenGL’s fragment processing pipeline

28. 28
Warped Lens Matched Scene
Warped version of Lens Matched Shading to match HMD lens

29. 29
Warped Lens Matched Shading

30. 30
Visualizing Warp Window Rectangles

31. 31
What Didn’t Have to be Warped
17% of left view pixels 17% of right view pixels

32. 32
VR Rendering Pipeline in Images
LMS Right Eye View Warped Right Eye View
LMS Left Eye View Warped Left Eye View
Scene
LMS = Lens Matched Shading

33. 33
VR Rendering Pipeline in Images
LMS Right Eye View Warped Right Eye View
LMS Left Eye View Warped Left Eye View
Scene
Displayed within HMD
Single Rendering Pass
Single Pass Stereo +
Lens Matched Shading +
Window Rectangle Testing
Drawn with Single Triangle
Fragment Shader Warping
Window Rectangle Testing
Perception to user is linear rendering
HMD lens “undoes” warping to provide
a perceived wide field-of-view
Pascal does all this
efficiently in a single
rendering pass!
8 viewports, 1 pass

34. 34
Traditional 3D Rendering Pipeline
Transform
3D View
Downsample
Display
Frame
App
Processing
if multisampled
uses pinhole camera projection,
rasterizing to uniform rectilinear pixel grid
monitor directly
displays uniform
rectilinear pixel
grid
Rasterize
3D View

35. 35
Virtual Reality Rendering Pipeline
Rendering Left & Right Eye Views
Transform
Left View
Downsample
Display
Frame to HMD
App
Processing
Rasterize
Left View
Transform
Right View
Downsample
Rasterize
Right View
HMD
Warp
HMD
Warp
warps uniform
rectilinear grid
based on HMD’s
device profile
Transform
Right View
Rasterize
Right View
Transform
Right View
Downsample
Rasterize
Right View
Transform
Right View
HMD
Warp
Downsample
Rasterize
Right View
Transform
Right View
second view rendering

36. 36
Virtual Reality Rendering Pipeline
With Lens Matched Shading
Transform
Left View
Downsample
Display
Frame to HMD
App
Processing
Rasterize
Left View
Transform
Right View
Downsample
Rasterize
Right View
HMD
Warp
HMD
Warp
Lens
Matched
Shading
Lens
Matched
Shading
replicates primitive
to 4 projected quadrants
less pixel resampling
effort & error operating on
LMS rasterized images

37. 37
Virtual Reality Rendering Pipeline
With Single Pass Stereo
Transform
Left & Right
View
Downsample
Display
Frame to HMD
App
Processing
Rasterize
Left View
Downsample
Rasterize
Right View
HMD
Warp
HMD
Warp
one transform instance,
outputs left & right position

38. 38
Virtual Reality Rendering Pipeline
Single Pass Stereo + Lens Matched Shading
Downsample
Display
Frame to HMD
Rasterize
Left View
Downsample
Rasterize
Right View
HMD
Warp
HMD
Warp
Lens
Matched
Shading
Lens
Matched
Shading
Transform
Left & Right
View
App
Processing
+
Advantages
1. Single transform for both views
2. Closer match between rendering & lens images
3. Fewer overall pixels for similar quality
use window rectangle testing
to bound lens region, during
both rasterization & warp

39. 39
Pascal GPU Features for VR
Unique to NVIDIA
Pascal renders left & right eye
In single rendering pass
Halves CPU load for rendering
Better concurrent texture locality
OpenGL: NV_stereo_view_rendering
Direct3D: NVAPI SinglePassStereoMode
Better matches rendering to HMD lens
Single pass
Four separately projected quadrants
Less rendering, less resampling error
OpenGL: NV_clip_space_w_scaling
Direct3D: NVAPI ModifiedWSupport

40. 40
Other NVIDIA Virtual Reality Initiatives

41. 41
More Information
VRWORKS 2.0 Software Development Kit
Available NOW
Pascal OpenGL extensions for Virtual Reality
Specification now public
In official OpenGL.org extension
registry
OpenGL Extension Wrangler 2.0 (GLEW) support
Example code in VRWORKS 2.0 SDK
VRWORKS also documents NVAPI usage
For Direct3D 11
Includes Examples code