Thèse "Enable the next Generation - Interactive video streaming" par Xavier CORBILLON, lors de la journée Futur & Ruptures du 31 janvier 2019. Une journée scientifique pour présenter l’ensemble des travaux de thèses aboutis portant sur des thématiques prospectives du programme de l’IMT.

1. Institut Mines-Télécom
IMT Atlantique
Bretagne-Pays de la Loire
École Mines-Télécom
Enable the Next Generation
Interactive Video Streaming
Xavier Corbillon
Thesis defended on
October 30, 2018
Rapporteurs: Vincent CHARVILLAT Professor University of Toulouse France
Miska M. HANNUKSELA Bell Labs Fellow Nokia Technologies Finland
Jury members:
President: Patrick LE CALLET Professor University of Nantes France
Examiners: Vincent CHARVILLAT Professor University of Toulouse France
Miska M. HANNUKSELA Bell Labs Fellow Nokia Technologies Finland
Laura TONI Lecturer University College London UK
Director: Gwendal SIMON Professor IMT Atlantique, IRISA France
Rapporteurs: Vincent CHARVILLAT Professo
Miska M. HANNUKSELA Bell Labs
Jury members:
President: Patrick LE CALLET Professo
Examiners: Vincent CHARVILLAT Professo
Miska M. HANNUKSELA Bell Labs
Laura TONI Lecturer
Director: Gwendal SIMON Professo

2. General Context: Video Streaming 2/12
Paradigm shift:
an area of on demand personalized delivery
Content Provider Internet
Pull
1
2
0
34 2
1
2
0
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

3. General Context: Video Streaming 2/12
Paradigm shift:
an area of on demand personalized delivery
Content Provider Internet
Pull
1
2
0
34 2
1
2
0
In 2016: 37 TB s−1 (67 % is video)
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

4. General Context: Video Streaming 2/12
Paradigm shift:
an area of on demand personalized delivery
Users request high quality
Content Provider Internet
Pull
1
2
0
34 2
1
2
0
In 2021: 105 TB s−1 (80 % is video)
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

5. How to Deliver Traditional Videos Today? 3/12
Main technology: HTTP Adaptive Streaming
• MPEG-DASH, Apple HLS, Adobe HDS, MSS
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

6. How to Deliver Traditional Videos Today? 3/12
Main technology: HTTP Adaptive Streaming
• MPEG-DASH, Apple HLS, Adobe HDS, MSS
Concept:
high
s1
med
low
s2 s3
tt
high
s1
med
low
s2 s3
server / CDN cache
Bandwidth PredictionSegment Generation
bw
t
client
connect
mpd
HTTP requests
s1:med
s2:high
s3:low
origin server
sent
once
Original Video
user
displayed
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

7. How to Deliver Traditional Videos Today? 3/12
Main technology: HTTP Adaptive Streaming
• MPEG-DASH, Apple HLS, Adobe HDS, MSS
Concept:
high
s1
med
low
s2 s3
tt
high
s1
med
low
s2 s3
server / CDN cache
Bandwidth PredictionSegment Generation
bw
t
client
connect
mpd
HTTP requests
s1:med
s2:high
s3:low
origin server
sent
once
Original Video
user
displayed
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

8. How to Deliver Traditional Videos Today? 3/12
Main technology: HTTP Adaptive Streaming
• MPEG-DASH, Apple HLS, Adobe HDS, MSS
Concept:
high
s1
med
low
s2 s3
tt
high
s1
med
low
s2 s3
server / CDN cache
Bandwidth PredictionSegment Generation
bw
t
client
connect
mpd
HTTP requests
s1:med
s2:high
s3:low
origin server
sent
once
Original Video
user
displayed
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

9. How to Deliver Traditional Videos Today? 3/12
Main technology: HTTP Adaptive Streaming
• MPEG-DASH, Apple HLS, Adobe HDS, MSS
Concept:
high
s1
med
low
s2 s3
tt
high
s1
med
low
s2 s3
server / CDN cache
Bandwidth PredictionSegment Generation
bw
t
client
connect
mpd
HTTP requests
s1:med
s2:high
s3:low
origin server
sent
once
Original Video
user
displayed
mpd = Manifest file
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

10. How to Deliver Traditional Videos Today? 3/12
Main technology: HTTP Adaptive Streaming
• MPEG-DASH, Apple HLS, Adobe HDS, MSS
Concept:
high
s1
med
low
s2 s3
tt
high
s1
med
low
s2 s3
server / CDN cache
Bandwidth PredictionSegment Generation
bw
t
med high low
client
connect
mpd
HTTP requests
s1:med
s2:high
s3:low
origin server
sent
once
Original Video
user
displayed
mpd = Manifest file
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

11. How to Deliver Traditional Videos Today? 3/12
Main technology: HTTP Adaptive Streaming
• MPEG-DASH, Apple HLS, Adobe HDS, MSS
Concept:
high
s1
med
low
s2 s3
tt
high
s1
med
low
s2 s3
server / CDN cache
Bandwidth PredictionSegment Generation
bw
t
med high low
client
connect
mpd
HTTP requests
s1:med
s2:high
s3:low
origin server
sent
once
Original Video
user
displayed
DASH design is challenged by new interactive contents
mpd = Manifest file
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

12. Omnidirectional Videos 4/12
O
#»
k
#»
#»ı
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

13. Omnidirectional Videos 4/12
O
#»
k
#»
#»ı
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

14. Omnidirectional Videos 4/12
O
#»
k
#»
#»ı
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

15. Omnidirectional Videos 4/12
O
#»
k
#»
#»ı#»u
#»v
Viewport
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

16. Omnidirectional Videos 4/12
O
#»
k
#»
#»ı
#»u
#»v
Viewport
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

17. Omnidirectional Videos Streaming 5/12
Thesis target: Stream on demand 360° videos to users’ devices
O
#»
k
#»
#»ı
Internet? ?
Storage
and
Content
Representation
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

18. High Feeling of Immersion 6/12
Constraints: High viewport resolution, low motion-to-photon delay
Viewport:
• 4K resolution in the viewport → 12K resolution for the full video
• at least 90 frames per second
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

19. High Feeling of Immersion 6/12
Constraints: High viewport resolution, low motion-to-photon delay
Viewport:
• 4K resolution in the viewport → 12K resolution for the full video
• at least 90 frames per second
Latency: Delay between action and visible result
• head moves without visible change → motion sickness
• ⇒ maximum 10 ms
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

20. High Feeling of Immersion 6/12
Constraints: High viewport resolution, low motion-to-photon delay
Viewport:
• 4K resolution in the viewport → 12K resolution for the full video
• at least 90 frames per second
Latency: Delay between action and visible result
• head moves without visible change → motion sickness
• ⇒ maximum 10 ms
2015 state of the art:
• send full 360-video requires >100 Mbit s−1
• send viewport only is not scalable and the latency is not low enough
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

21. Quality Emphasized Region 7/12
Traditional videos are encoded with uniform quality
Encoding based:
Uniform Quality
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

22. Quality Emphasized Region 7/12
360° videos can be prepared with non uniform spatial quality
Encoding based:
Quality Emphasized Region
good degradation
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

23. Quality Emphasized Region 7/12
360° videos can be prepared with non uniform spatial quality
Encoding based, Projection based, Region wise packing
Quality Emphasized Region
good degradation
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

24. Viewport Adaptive Streaming 8/12
Viewport adaptive streaming
high
s1
low
QER1
high
low
QER2
high
low
QER3
s2 s3
t
server
bw
t
low high low
client
connect
mpd
s1:QER2 lo
s2:QER3 hi
s3:QER1 lo
QER = Quality Emphasized Region
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

25. Viewport Adaptive Streaming 8/12
Viewport adaptive streaming
high
s1
low
QER1
high
low
QER2
high
low
QER3
s2 s3
t
server
bw
t
low high low
client
connect
mpd
s1:QER2 lo
s2:QER3 hi
s3:QER1 lo
QER = Quality Emphasized Region
Publication:
X. Corbillon, A. Devlic, G. Simon, and J. Chakareski, “Viewport-adaptive navigable 360-degree video delivery”,
in IEEE International Conference on Communications (ICC), 2017, pp. 1–7 Best Paper Award
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

26. “Quality” VS Number of QER 9/12
Optimal QER generation for omniscient clients
1 2 3 4 5 6 7 8 9 10 11
0.5
1
1.5
2
surface bit-rate for a uniform quality
Number of QER versions offered
Visible
surfacebit-rate[Mbits−1
m−2
]
roller-coaster
diving
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

27. “Quality” VS Number of QER 9/12
Optimal QER generation for omniscient clients
1 2 3 4 5 6 7 8 9 10 11
0.5
1
1.5
2
surface bit-rate for a uniform quality
Number of QER versions offered
Visible
surfacebit-rate[Mbits−1
m−2
]
roller-coaster
diving
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

28. “Quality” VS Number of QER 9/12
Optimal QER generation for omniscient clients
1 2 3 4 5 6 7 8 9 10 11
0.5
1
1.5
2
surface bit-rate for a uniform quality
Number of QER versions offered
Visible
surfacebit-rate[Mbits−1
m−2
]
roller-coaster
diving
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

29. Takeaway 10/12
Thesis Target: Stream on demand 360° videos to users’ devices
Contributions:
Recorded one of the first Head Movement dataset
Propose a Viewport Adaptive Streaming architecture based on QERs
Propose a model to generate optimal set of QERs
Extend the evaluation of Facebook’s offset projections
Extend Viewport Adaptive Streaming for multi-viewpoint 360° videos
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

30. Future Work 11/12
How to integrate Quality Emphasized Regions (QERs) in live streaming?
• Automatic saliency estimation
• Distributed computation to feed the origin servers with relevant features
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

31. Future Work 11/12
How to integrate Quality Emphasized Regions (QERs) in live streaming?
• Automatic saliency estimation
• Distributed computation to feed the origin servers with relevant features
We studied static QERs; Are dynamic QERs more efficient?
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

32. Future Work 11/12
How to integrate Quality Emphasized Regions (QERs) in live streaming?
• Automatic saliency estimation
• Distributed computation to feed the origin servers with relevant features
We studied static QERs; Are dynamic QERs more efficient?
Caching optimization for / with QERs
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

33. Future Work 11/12
How to integrate Quality Emphasized Regions (QERs) in live streaming?
• Automatic saliency estimation
• Distributed computation to feed the origin servers with relevant features
We studied static QERs; Are dynamic QERs more efficient?
Caching optimization for / with QERs
Light field / Point Cloud compression for real 6DoF videos
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

34. Publications 12/12
International Conferences:
2018
• H. Hristova, X. Corbillon, G. Simon, V. Swaminathan, and A. Devlic, “Heterogeneous spatial quality for omnidirectional video”, in
Proceeding of IEEE International Workshop on Multimedia Signal Processing (MMSP), 2018 Best Paper Award
• X. Corbillon, F. De Simone, G. Simon, and P. Frossard, “Dynamic adaptive streaming for multi-viewpoint omnidirectional videos”,
in Proceedings of the 9th ACM Multimedia Systems (MMSys), 2018, pp. 237–249 Best Paper Award
• J. Chakareski, R. Aksu, X. Corbillon, and G. Simon, “Viewport-driven rate-distortion optimized 360° video streaming”, in
Proceedings of IEEE International Conference on Communications (ICC), 2018, pp. 1–7
2017
• X. Corbillon, A. Devlic, G. Simon, and J. Chakareski, “Optimal set of 360-degree videos for viewport-adaptive streaming”, in
Proceedings of ACM on Multimedia Conference, MM, 2017, pp. 943–951
• X. Corbillon, F. De Simone, and G. Simon, “360-degree video head movement dataset”, in Proceedings of the 8th ACM on
Multimedia Systems Conference (MMSys’17), 2017, pp. 199–204
• X. Corbillon, A. Devlic, G. Simon, and J. Chakareski, “Viewport-adaptive navigable 360-degree video delivery”, in IEEE
International Conference on Communications (ICC), 2017, pp. 1–7 Best Paper Award
2016
• X. Corbillon, F. Boyrivent, G. A. D. Williencourt, G. Simon, G. Texier, and J. Chakareski, “Efficient lightweight video packet
filtering for large-scale video data delivery”, in IEEE International Conference on Multimedia & Expo Workshops, ICME
Workshops, 2016, pp. 1–6
• X. Corbillon, R. Aparicio-Pardo, N. Kuhn, G. Texier, and G. Simon, “Cross-layer scheduler for video streaming over MPTCP”, in
Proceedings of the 7th International Conference on Multimedia Systems (MMSys’16), 2016, 7:1–7:12
Journals:
2018
• M. Ben Yahia, Y. Le Louedec, G. Simon, L. Nuaymi, and X. Corbillon, “HTTP/2-based frame discarding for low-latency adaptive
video streaming”, IEEE Transactions on Multimedia Computing Communications and Applications, 2018, Accepted for publication
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

35. Submitted Work 12/12
Journals:
2018
• X. Corbillon, H. Hristova, A. Devlic, and G. Simon, “Survey on omnidirectional videos streaming”, IEEE Communications Surveys
& Tutorials, 2018, in submission
• J. Liu, G. Simon, X. Corbillon, J. Chakareski, and Q. Yang, “Mobile edge caching for viewport-adaptive 360-degree video
streaming”, IEEE Transactions on Circuits and Systems for Video Technology, 2018, in submission
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

36. Collaborations 12/12
Francesca De Simone & Pascal Frossard (visiting PhD)
Jacob Chakareski
Alisa Devlic
Vishwanathan Swaminathan
Jiayi Liu
Hristina Hristova & Mariem Ben Yahia
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

37. Questions 12/12
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming

38. Takeaway 12/12
Thesis Target: Stream on demand 360° videos to users’ devices
Contributions:
Recorded one of the first Head Movement dataset
Propose a Viewport Adaptive Streaming architecture based on QERs
Propose a model to generate optimal set of QERs
Extend the evaluation of Facebook’s offset projections
Extend Viewport Adaptive Streaming for multi-viewpoint 360° videos
IMT Atlantique January 31, 2019 Xavier Corbillon Enable the Next Generation Interactive Video Streaming