Verizon Labs developed an augmented and virtual reality platform called Envrmnt using AWS GPU instances to overcome challenges with real-time video rendering. They achieved a 10x speed improvement for stitching and encoding 360 degree video by leveraging GPU parallelization. The new architecture processes video in real-time using GPUs for decoding, stitching and 2D/3D rendering, while encoding is performed efficiently on CPUs. This allows AR and VR content to be streamed seamlessly to any device.

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
November 30, 2016
CMP312
Powering the Next Generation
of Virtual Reality with Verizon
Puneet Agarwal, AWS Solutions Architect
Christian Egeler, Director AR/VR, Verizon Labs
Vinay Polavarapu, Lead Architect, Verizon Labs

2. AWS GPU Instances

3. EC2 instance families
General Purpose: M1, M3, M4, T2
Compute Optimized: C1, CC2, C3, C4
Memory Optimized: M2, CR1, R3, X1
Storage Optimized: HI1, HS1, I2, D2
GPU: CG1, G2, P2
Micro: T1, T2

4. New P2 GPU instance types
• Offers up to 16 NVIDIA K80 GPUs (8 K80 cards) in a
single instance
• The 16xlarge size provides:
• A combined 192 GB of GPU memory, 40 thousand CUDA cores
• 70 teraflops of single precision floating point performance
• Over 23 teraflops of double precision floating point performance

5. P2 instance types
Instance
Size
GPUs GPU Peer
to Peer
vCPUs Memory
(GiB)
Network
Bandwidth*
p2.xlarge 1 - 4 61 1.25Gbps
p2.8xlarge 8 Y 32 488 10Gbps
p2.16xlarge 16 Y 64 768 20Gbps
*In a placement group

6. Why GPUs?
• GPUs can perform certain functions more efficiently than
software running on CPUs
• Each GPU has thousands of processing cores, enabling
• High degree of parallelism and throughput
• Low latencies
• Lower cost
• Power efficiency

7. GPU for accelerated computing
for (i=0;i<N;i++) {
}
…
for (j=0;j<M;j++) {
}
GPU handles compute-
intensive functions
5% of code
80% of run-time
CPU handles the rest

8. GPU use cases
• Machine learning
• Engineering simulation
• Financial simulation
• Virtual reality
• Database
• Rendering
• Transcoding
• And many more…

9. Samsung R&D Institute Russia Uses AWS to Speed Up
Research by 50%
Samsung R&D Institute Russia is one of Samsung
Electronics’ leading advanced research centers.
Thanks to AWS, we were able to
reduce the time of development of
state-of-the-art Machine Learning
models by almost half
Mikhail N. Rychagov, Dr.Sc., Director
Algorithm Lab,
Samsung R&D Institute Russia
”
“
• Creating machine learning algorithms and
applications requiring intense compute
resources
• Using AWS to run GPU clusters that
scale depending on fluctuating demand
• Gained flexibility and scalability
• Can now build recognition and
classification engines for intelligent data
processing tasks
• Costs for support and maintenance of
infrastructure were significantly reduced

10. Augmented and Virtual Reality
with Verizon

11. Who are we?
Christian Egeler
Director of Augmented and Virtual Reality Product Development at Verizon.
A telecommunications and device software platform veteran with industry
experience around public safety, OTT VoIP, Image Recognition, OS
Platform, and Ultra Wideband. Currently focused on how AR and VR will
contribute and integrate into the next compute platform.
Vinay Polavarapu
Architect and developer of cloud technologies for the video platform at
Yahoo. Now leveraging that experience to architect and build critical
components for the AR/VR video platform at Verizon Labs.

12. Today we will talk about…
• Challenges in developing AR & VR technology
• How Verizon Labs has used AWS GPUs to overcome
some key technical challenges
• Initial Architecture
• Improved Architecture
• Demo!

13. All of the things we do…
wireless fios enterprise streaming CDN
big data security communications
advertising internet of things monitoring

14. And with the AOL acquisition…
advertising digital media
lifestyles entertainment news

15. Why Verizon is positioned to develop AR/VR
Power dozens of devices and services with an evolved network
and set of service platforms
Helped businesses innovate and get their products to consumers
Verizon serves as the bridge between advertising, content, streaming,
retail and the actual network(s) which makes us uniquely poised to create
an ecosystem for AR/VR creation and distribution

16. And the vision?
As the world gets closer to AR and VR, the industry find
ourselves needing to master rendering and computer vision in
the cloud and on ever smaller form factors like headsets.
This brings new demands to silicon and the network.
“It is potentially the final computing platform. This is generalizing, but if you have perfect virtual
reality, you don't have to perfect much else.”
Palmer Luckey, Oculus founder

17. How did Verizon Labs tackle
this challenge?
17
• A cutting edge delivery platform for streaming AR and VR experiences
• Help users create interactive 3D content with easy to use tools
• Live-stream AR & VR experiences with multi-user support for interaction
• Monetize content and connect advertisers with consumers through relevant interactive
ads both on the smartphone and headset

18. What does Envrmnt enable?
18
• Accommodate all capture solutions and media
• Live-stream and live-stitch rich and immersive content
• Build worlds and connect your audience
• Projection or augmentation of content
• Bring digital advertising into AR and VR
• View your AR/VR content anywhere
• Deliver it fast and seamlessly

19. Why we chose to build on AWS?
• Infrastructure as service model
• CapEx- zero initial capital expenditure
• Scalability and elasticity - scaling up and scaling down is few clicks away
• Cost efficiency – Pay-as-you-go model helps save money when servers are not in
use
• Flexibility - Use any cloud architecture, operating systems, databases, languages or
tools
• Security - AWS complies with highest industry standard security practices, one less
thing to worry about

20. What we built?
• End to End VR content creation, hosting and delivery platform
• VR Content authoring Service for ingest
• Media Library service for content consumption
• Ads Service for ad integration and monetization
• Real-time Stitching and Encoding Service for live and on-demand ultra
HD VR content from a variety of cameras and rigs
• Adaptive streaming capability to stream to any device using HLS and
MPEG-DASH .
• Mobile VR rendering platform for IOS and Android

21. Stitching inputs

22. Stitching Output

23. Availability Zone #2Availability Zone #1
What we built
Elastic Load
Balancing
Amazon S3
bucket
CDN
EC2 instance
Web/App
server
Amazon
Route 53
CDN
CDN
CDN
Elastic Load
Balancing
Elastic Load
Balancing
Elastic Load
Balancing
Auto Scaling group
EC2 instance
Web/App
server
EC2 instance
Web/App
server
EC2 instance
Web/App
server
Wowza EC2
Auto Scaling group
Wowza EC2 Wowza EC2 Wowza EC2
Elastic Load
Balancing
CDN
RDS Aurora Master RDS Aurora Read Replica
Player Customers
Marketing Users
Authoring Users
Web Player Users
Image Users
Encoding and
Stitching EC2
Environment

24. Challenges
Augmented and Virtual Reality has a base requirement
of low latency render, compute and network
transmission
We now have a technological dependency on high resolution
rendering, high refresh rate and loads of computer vision.
Low resolution and lag are no longer just annoying, they are a deal
breaker

25. Challenges
There is a fundamental need on the network and the GPU to handle rending
and compute for the foreseeable future
Source: blogs.nvidia.com
Source: ericsson 2014

26. How can we solve these problems?

27. Real-time video rendering
27
• GPU providers are doing their
part
• Using AWS GPUs in the
cloud, we have enhanced our
platform to stitch, encode, and
stream AR and VR
experiences at near real-time
speed

28. Live stitching
28
Our stitching pipeline utilized CUDA and
GPU parallelization in order to optimize our
stitching pipeline
4x decoding speed
10x speed for warping, blending and un-
distortion in comparison to CPUs
10x overall performance gain vs. similar
class CPU

29. Real time 360 Video Stitching
29
0
28
55
83
110
138
m4.16xlarge g2.8xlarge p2.16xlarge
Stitching frames per second
Instance Types

30. Encoding
30
• Currently, CPU-based encoding is performing better for
us than GPU-based encoding
• Our peers in the industry claim that they get better
encoding performance on GPUs
• We are experimenting with both CPUs and GPUs in a
hybrid setup to get real-time encoding speed for 4K
content

31. Real time stitching and encoding
Real-Time Auto-Stitching
is performed
with very little to no
manual intervention
utilizing CUDA
Elastic Load
Balancing
CloudFront
Distribution
CloudFront
Distribution
CloudFront
Distribution
Elastic Load
Balancing
Elastic Load
Balancing
VR Ready Content
Raw Switchable
Footage
Live Event
Player Users
Authoring and
Processing
Authoring and
Processing
Decoding and
stitching on GPU
Cluster
Encoding on CPU
Cluster
Source S3
bucket
Output S3
bucket
Media Servers
Real-Time/Live Encoding
and Stitching

32. Benefits of new architecture
• Ability to leverage both GPUs and CPUs to perform tasks
as per their strength
• 2D and 3D rendering in real time on CUDA cores on
GPUs
• Better decoding and stitching function speeds using
hardware decoder on GPUs
• Encoding is more efficient on CPUs
• Achieved real-time processing speeds!

33. Where did we take this technology?
33
• Augmented and Virtual Reality that is
streamed to any platform
• Easy-to-use tools that allow anyone to set
up and create a scene
• Monetization through the ad provider of
your choice
• Direct access to your audience with
detailed analytics

34. And now for the Demos!
34

36. 36

37. What we wish we knew…
• Explore GPUs before CPUs for real-time video rendering
• For some low level encoding tasks, CPUs may work
better
• Leverage Auto Scaling

38. Where do we think there is still work to be done?
38
As network technology develops on, it becomes more of a
possibility to offload GPU compute tasks to the cloud.
Future enhancements of low latency high bandwidth
connections should consider optimizations for graphics
processing.

39. So to recap!
39
• A cutting edge delivery platform for streaming AR and VR experiences
• Help users create interactive 3D content with easy to use tools
• Live-stream AR & VR experiences with multi-user support for interaction
• Monetize content and connect advertisers with potential consumers through relevant
interactive ads both on the smartphone and headset

40. Thank you!
40
• Contact Christian Egeler
• Email – Christian.Egeler@VerizonWireless.com
• Contact Vinay Polavarapu
• EMail – vinay.polavarapu@envrmnt.com
We’re Hiring!
• Video Encoding Algorithm Architect
• 3D Computer Vision and SLAM Engineers
• Volumetric Reconstruction Engineers
• Numerical Mathematicians / Algorithms Engineers
• CUDA Experts
• OpenGL Rendering Experts
• Located in NJ, NY, & San Francisco with satellite teams all over the US!

41. Thank you!

42. Remember to complete
your evaluations!