The Virtual Reality (VR) is considered by industrials from content industry as a major technology to develop in the next years. It comes however with a number of challenges, which will require the cooperation between multiple actors in the content delivery chain. Since it combines high quality multimedia delivery and low-latency interactivity, VR matches the requirements of 5G networks and it has the potential to be a key driver for adoption of the next generation network. In this talk, the main requirements of the envisioned next-generation VR applications will be reviewed, especially the need of both bandwidth and latency. Then, the main delivery architectures will be presented, including their main weaknesses in today’s networks and the efforts that are currently done in standardization groups to provide the main elements of these architectures in the perspective of 5G. Finally, a selection of the main open challenges will conclude the talk.

1. IMT Atlantique
Bretagne-Pays de la Loire
École Mines-Télécom
Virtual Reality in 5G Networks:
Solutions and Open Problems
Gwendal SIMON

2. Simultaneous Releases of Key Technologies 2/16
omnidirectional
cameras
panoramic videos immersive devices
IMT Atlantique Jun 2017 Gwendal SIMON Virtual Reality in 5G Networks

3. Classification of Expected Applications 3/16
Type Examples Implementation
360° scenes reacting
to user command
gaming, 6DoF video,
training simulation
game engine close to the
end-users
passive navigable
360° scene (3DoF)
streaming, adult, sport,
documentaries
Content Delivery Network
(CDN)
live navigable 360°
one-to-one call
emergencies, visiophone,
online assistance
an integrated delivery
chain including proxies
IMT Atlantique Jun 2017 Gwendal SIMON Virtual Reality in 5G Networks

4. Requirements 4/16
Bandwidth: bit-rate necessary to transport 360° multimedia flow
• 4k resolution in the viewport → 16k resolution for the full video
• at least 90 frames per second
⇒ minimum 150 Mbps
Latency: delay between action and visible result
• head moves without visible change → motion sickness
⇒ maximum 10 ms
• gamer command to impact on game world → bad gaming QoE
⇒ maximum 70 ms?
Computing Resources: configurations for servers to prepare flow
• latest generation GPU cards + huge memory + multi-core machine
IMT Atlantique Jun 2017 Gwendal SIMON Virtual Reality in 5G Networks

5. Gaming
IMT Atlantique Jun 2017 Gwendal SIMON Virtual Reality in 5G Networks

6. Today: Arcade Rooms 6/16
IMT Atlantique Jun 2017 Gwendal SIMON Virtual Reality in 5G Networks

7. Cloud VR Gaming: The Chain of Latency 7/16
Response time is the delay from an ac-
tion done at the user and the result of
this action on the display
IMT Atlantique Jun 2017 Gwendal SIMON Virtual Reality in 5G Networks

8. Cloud VR Gaming: The Chain of Latency 7/16
gateway
t1
Internet
t2
datacenter
t3 user input
processing
t4
game state
update
t5
graphics
rendering
t6
video
encoding
t7
network
delivery
t1 + t2 + t3video
decoding
t8
latency due to network
latency due to game engine
latency due to video encoding and decoding
IMT Atlantique Jun 2017 Gwendal SIMON Virtual Reality in 5G Networks

9. Cloud VR Gaming: Reality Check 8/16
Latency due to game engine:
• ranges from 10 ms to 25 ms (with state-of-the-art GPU-powered servers)
Latency due to video encoding and decoding
• ranges from 20 ms to 40 ms (with state-of-the-art hardware encoders)
Latency due to the network
• must range from 5 ms to 20 ms to reach a full-featured datacenter
Toward the new generation of distributed game engines
where modules are spread into latency-optimized networks
IMT Atlantique Jun 2017 Gwendal SIMON Virtual Reality in 5G Networks

10. Streaming
IMT Atlantique Jun 2017 Gwendal SIMON Virtual Reality in 5G Networks

11. Viewport-Adaptive Streaming 10/16
Existing (and considered) solutions
• Stream the whole 360-degree video
Server Client
extracted viewport
viewport resolution bandwidth
HD 6K >50 Mbps
UHD 8K >80 Mbps
4K 16K >150 Mbps
Advantages:
• Interactive
• DASH compliant today
Weaknesses:
• Bandwidth consumption
IMT Atlantique Jun 2017 Gwendal SIMON Virtual Reality in 5G Networks

12. Viewport-Adaptive Streaming 10/16
Existing (and considered) solutions
• Stream only the user viewport
Server
extracted viewport
Client
position
Action Latency
Request to the server RTT/2
Viewport extraction 5 ms
Re-encoding 10 ms
Transmission RTT/2
Decoding 7 ms
Advantages:
• Bandwidth optimization
Weaknesses:
• Computation at the server
IMT Atlantique Jun 2017 Gwendal SIMON Virtual Reality in 5G Networks

13. Viewport-Adaptive Streaming 10/16
Existing (and considered) solutions
• Split the 360-degree video into independent tiles selected by the client
Server Client
Selection
Advantages:
• Good interactivity
• Flexibility
• Compliant with DASH SRD
Weaknesses:
• Client side processing
• Storage at server
• Compression efficiency
IMT Atlantique Jun 2017 Gwendal SIMON Virtual Reality in 5G Networks

14. Viewport-Adaptive Streaming 10/16
Our proposal:
X. Corbillon, A. Devlic, J. Chakareski, and G. Simon “Viewport-Adaptive Navigable
360-Degree Video Delivery ” , in IEEE ICC, 2017 (best paper award)
X. Corbillon, F. de Simone, and G. Simon. “360-Degree Video Head Movement
Dataset ”, in ACM MMSys (Dataset track), 2017
• Introduce the notion of Quality Emphasis Region (QER)
• Predict the head movements
• Optimize video encoding by selecting the best QERs
Toward ultra-low-latency streaming with full cooperation between network
and application layers as well as better models to predict user behavior
IMT Atlantique Jun 2017 Gwendal SIMON Virtual Reality in 5G Networks

15. Standards
IMT Atlantique Jun 2017 Gwendal SIMON Virtual Reality in 5G Networks

16. Standardization Efforts: OMAF at MPEG 12/16
Equirectangular
IMT Atlantique Jun 2017 Gwendal SIMON Virtual Reality in 5G Networks

17. Standardization Efforts: OMAF at MPEG 12/16
Equirectangular with 8 × 8 tiles packed with different resolutions
IMT Atlantique Jun 2017 Gwendal SIMON Virtual Reality in 5G Networks

18. Standardization Efforts: OMAF at MPEG 12/16
4 × 3 cubemap
IMT Atlantique Jun 2017 Gwendal SIMON Virtual Reality in 5G Networks

19. Standardization Efforts: OMAF at MPEG 12/16
Compact cubemap
IMT Atlantique Jun 2017 Gwendal SIMON Virtual Reality in 5G Networks

20. Standardization Efforts: OMAF at MPEG 12/16
Pyramid
IMT Atlantique Jun 2017 Gwendal SIMON Virtual Reality in 5G Networks

21. Standardization Efforts: OMAF at MPEG 12/16
Compact pyramid
IMT Atlantique Jun 2017 Gwendal SIMON Virtual Reality in 5G Networks

22. Standardization Efforts: OMAF at MPEG 12/16
Compact rhombicdodecahedron
IMT Atlantique Jun 2017 Gwendal SIMON Virtual Reality in 5G Networks

23. Standardization Efforts: DASH-VR at MPEG 13/16
Ongoing work under discussion:
Signalling a Region of Interest (RoI)
Sharing viewing statistics to enable accurate head movement prediction
Interfacing with Server and Network-Assisted Delivery (SAND)
high
s1
low
QER1
high
low
QER2
high
low
QER3
s2 s3
t
server
bw
t
low high low
client
connect
mpd
s1:QER2 lo
s2:QER3 hi
s3:QER1 lo
IMT Atlantique Jun 2017 Gwendal SIMON Virtual Reality in 5G Networks

24. Standardization Efforts: Protocols 14/16
WebRTC at W3C: no specific action for 360° video yet
Congestion control for high-bandwidth and low-latency videos
HTTP/2 at IETF: various improvements for low-latency videos
Server push to anticipate requests from clients
Multiplexing: priority and cancelling of video parts
M. Ben Yahia, Y. Le Louedec, L. Nuaymi, and G. Simon. “When HTTP/2 Rescues DASH: Video
Frame Multiplexing ” IEEE Infocom Workshop, 2017.
MPTCP and QUIC at IETF: dealing with transport layer issues
Cross-layer approaches to avoid useless retransmission
X. Corbillon, R. Aparicio-Pardo, N. Kuhn, G. Texier, and G. Simon. “Cross-Layer Scheduler for Video
Streaming over MPTCP ” ACM MMSys, 2016.
IMT Atlantique Jun 2017 Gwendal SIMON Virtual Reality in 5G Networks

25. Conclusion
IMT Atlantique Jun 2017 Gwendal SIMON Virtual Reality in 5G Networks

26. Takeway and Conclusion 16/16
Virtual Reality: hype or trend?
Interactive applications based on very high-quality multimedia content
Use-cases that meet 5G requirements. . . two years too early
Latency is key: be smart from transport to application layers
Orchestrate in-network GPU servers with latency-targeting tasks
Collaborate with service providers to fill the gap between QoS and QoE
Predict behaviors to anticipate needs
IMT Atlantique Jun 2017 Gwendal SIMON Virtual Reality in 5G Networks