In this contribution, we present selected novel approaches and results of our research work in the \ATHENA Christian Doppler Laboratory (Adaptive Streaming over HTTP and Emerging Networked Multimedia Services), a major research project at our department jointly funded by public sources and industry. By putting this work also into the context of related ongoing research activities, we aim at working out where HTTP Adaptive Streaming is currently heading.

1. HTTP Adaptive Streaming –
Where Is It Heading?
Christian Timmerer, Assoc.-Prof. at AAU, CIO at Bitmovin
Hermann Hellwagner, Professor at AAU
Klagenfurt, Austria
November 30th
, 2020
1

2. “By 2022, Internet video will represent
82% of all Internet traffic.”
Cisco Visual Networking Index: Forecast and Trends, 2017–2022 (White Paper), Cisco,
February 2019.
2

3. ● Introduction
● ATHENA
○ Content Provisioning
○ Content Delivery
○ Content Consumption
○ End-to-End Aspects
○ Quality of Experience
● Conclusions: HAS - Where Is It Heading?
Agenda
3

4. Presenters
Christian Timmerer
Assoc.-Prof at
Alpen-Adria-Universität Klagenfurt
CIO | Head of Research and
Standardization at Bitmovin
Hermann Hellwagner
Professor at
Alpen-Adria-Universität Klagenfurt
4
4

5. Video streaming is dominating today’s Internet traffic
● May 2020: 57.64%; YouTube is the undisputed king with
15.94% followed by Netflix with 11.42%*
● By 2022, video will account for 82% of global IP traffic and
live video will increase 15-fold and reach 17% of Internet
video traffic**
Motivation
Sources: * Sandvine Global
Internet Phenomena (May 2020).
** Cisco Visual Networking Index
(VNI), Complete Forecast Update,
2017–2022 (Dec. 2018)
5
5

6. HTTP Adaptive Streaming 101
Adaptation logic is within the
client, not normatively specified
by a standard, subject to
research and development
6
6
Client

7. “Application-oriented basic research” to address current and future research
and deployment challenges of HAS and emerging streaming methods
ATHENA - Adaptive Streaming over HTTP and
Emerging Networked Multimedia Services
Content Provisioning Content Delivery Content Consumption
End-to-End Aspects
● Video encoding for HAS
● Quality-aware encoding
● Learning-based encoding
● Multi-codec HAS
● Edge computing
● Information CDN/SDN⇿clients
● Netw. assistance for/by clients
● Utility evaluation
● Bitrate adaptation schemes
● Playback improvements
● Context and user awareness
● Quality of Experience (QoE) studies
● Application/transport layer enhancements
● Quality of Experience (QoE) models
● Low-latency HAS
● Learning-based HAS
https://athena.itec.aau.at/
7
7
Funding:

8. Multimedia Systems Challenges and Tradeoffs
8
8
Basic figure by Klara Nahrstedt

9. Video coding for HTTP Adaptive Streaming
● Quality improvement
Per-Title Encoding (PTE) et al.: per-title/scene/shot/segment, content-/context
aware, content-adaptive, quality-aware encoding
● Runtime improvement
Hardware-/software-based (cloud), parallel/distributed, information reuse from
reference encodings (multi-rate/-resolution)
● Application scenarios
Video on Demand (VoD incl. diff. flavors AVoD, SVoD), live (incl. diff. flavors),
interactive, games, video conferencing
Content Provisioning
9
9

10. Video Encoding
Block
Partitioning
Motion
Compensation
Transformation
& Quantization
Entropy Coding
Entropy
Decoding
Inverse
Transformation &
Inverse
Quantization
Inter or Intra
Prediction
Picture Buffer In-loop Filtering
10

11. Video Encoding with Machine Learning
11
11
Block
Partitioning
Motion
Compensation
Transformation
& Quantization
Entropy
Coding
Entropy
Decoding
Inverse
Transformation &
Inverse Quantization
Inter or Intra
Prediction
Picture
Buffer
In-loop
Filtering
CTU
Decision
Prediction
Optical Flow
Detection
Mode
Prediction
Angular
Direction
Prediction
Deblocking
with ML
Denoising
with ML
Super-resolution

12. 1)
D. Schroeder et al. "Efficient multi-rate video encoding for HEVC-based adaptive HTTP streaming."
IEEE Transactions on Circuits and systems for Video Technology 28.1 (2016): 143-157.
2)
B. Guo, Y. Han, J. Wen, "Fast Block Structure Determination in AV1-based Multiple Resolutions Video
Encoding," 2018 IEEE Int’l. Conf. on Multimedia and Expo (ICME), San Diego, CA, USA, July 2018.
3)
H. Amirpour, E. Çetinkaya, C. Timmerer and M. Ghanbari, "Fast Multi-rate Encoding for Adaptive
HTTP Streaming," Data Compression Conference (DCC), Snowbird, UT, USA, 2020,
Encoding information can be used among different
quality representations
State-of-the-art:
● Encode the highest quality1)
or the lowest quality2)
as the
reference first, then use this information
Proposed method:
● Encode the highest quality first, then use its information to
encode the lowest quality and then use information from
both representations to encode the remaining
representations3)
● Double bound for CTU search ranges
Fast Multi-rate Encoding
12
12
QP1
QPN
QPN-1QP3
QP2
...
Encodingruntime(norm.)

13. Parallel encoding is still problematic
State-of-the-art:
● Encode the highest quality1)
or the lowest quality2)
as the
reference first, then use this information
Proposed method:
● Try different quality levels as the reference representation to
determine the best starting point for parallel encoding
● Encode the middle quality first, then use its information to
reduce the time-complexity for higher qualities to eliminate
possible bottlenecks3)
● Upper or lower bound depending on the quality level
Fast Multi-rate Encoding
1)
Schroeder et al., 2016 2)
Guo et al., 2018
3)
H. Amirpour, E. Çetinkaya, C. Timmerer and M. Ghanbari, "Towards Optimal
Multirate Encoding for HTTP Adaptive Streaming," International MultiMedia
Modeling Conference (MMM), Prague, Czech Republic, 2021
QPN/2
QPN
QP2
QP1
...
13
13
Encodingruntime(norm.)

14. Use ML to encode dependent representations
State-of-the-art:
● Use a CNN to predict CTU depth decisions1)
Proposed method:
● Train a CNN with encoding information obtained from
the reference quality (the lowest quality) representation
and use its decision to encode dependent
representations2)
● Focus on parallel encoding, thus only apply for
bottleneck situations
● Train different CNNs for different QP targets
Fast Multi-rate Encoding
with Machine Learning (FaME-ML)
1)
Kim, Kyungah, and Won Woo Ro. "Fast CU depth decision for HEVC using neural networks."
IEEE Transactions on Circuits and Systems for Video Technology 29.5 (2018): 1462-1473.
2)
E. Çetinkaya, H. Amirpour, C. Timmerer and M. Ghanbari, “FaME-ML: Fast Multirate
Encoding for HTTP Adaptive Streaming Using Machine Learning,” 2020 IEEE International
Conference on Visual Communications and Image Processing (VCIP), Macau, 2020
QPN
CNN
QPN-1
QP1
QP2
...
HEVC
HEVCHEVC
CNN
HEVC HEVC
14
14
Encodingruntime(norm.)

15. End Game
ML
Encoder
ML
Decoder
15
15
See also:
● CLIC: Workshop and Challenge on Learned Image Compression, https://www.compression.cc/
● JPEG AI-based image coding, https://jpeg.org/
● JVET (MPEG/VCEG) AI-based video coding, http://mpeg.org/

16. Network assistance for HTTP Adaptive Streaming
● Edge computing support (at CDN / cellular network edge)
Functions at (or, assisted by) the edge: adaptation, analytics, (pre-)fetching,
caching, transcoding, repackaging of content, request aggregation
● Server/network/CDN ↔ HAS client information exchange and collaboration
IETF ALTO, MPEG SAND, MPEG NBMP, …; SDN-DASH, SDN-HAS, SABR, ...
● Use of modern network architecture features
SW Defined Networking (SDN); Network Function Virtual. (NFV); MC-ABR
● Low-latency live streaming
Use of MPEG CMAF, HTTP/1.1 Chunked Transfer Encoding (CTE), other protocol
features (e.g., HTTP/2 Push); LL-HLS; specific network functions; CDN support
Content Delivery
16
16

17. Approach:
● Deliver CMAF segments only over the core/CDN
● Repackage into requested format at the edge
Evaluation:
● Analytical model and simulation to assess bandwidth
savings as compared to all-formats delivery (~ 20%)
● Measurements to get segment repackaging times
(CMAF➝HLS: 45-67 ms, depending on seg. size)
● Real world-like testbed to assess “full” repackaging time
(avg. 136 ms, CAdViSE on AWS cloud, 4-sec. seg./1080p)
Dynamic Segment Repackaging at the Edge for HAS
17
17
Jesus Aguilar-Armijo, Babak Taraghi, Christian Timmerer, and Hermann Hellwagner.
”Dynamic Segment Repackaging at the Edge for HTTP Adaptive Streaming”.
IEEE Int’l. Symposium on Multimedia (ISM'20). Dec. 2020.

18. Approach:
● Employ SDN and NFV concepts to mitigate
Multicast ABR problems
● SDN: to set up and optimize multicast paths
● VRPs (Virtual Proxies): to aggregate clients’ requests
● VTFs (Virtual Transcoders): to transcode segments
to quality levels requested by clients
● MILP optimization model to jointly construct
optimal multicast tree and VTFs placement
● Heuristic algorithm (polynomial time)
On Optimizing Resource Utilization in Live Video Streaming
18
18
Alireza Erfanian, Farzad Tashtarian, Reza Farahani, Christian Timmerer, and Hermann Hellwagner.
”On Optimizing Resource Utilization in AVC-based Real-time Video Streaming”.
IEEE Conf. on Network Softwarization (NetSoft'20). June/July 2020.

19. Example: MC-ABR OSCAR OSCAR (VTFs closer to edge)
On Optimizing Resource Utilization in Live Video Streaming
19
19
37 Mbps
37 Mbps

20. Sample results:
On Optimizing Resource Utilization in Live Video Streaming
20
20
Bandwidth/#OFcommands(norm.)
Comparison of proposed algorithm (heuristic) and
other approaches in terms of bandwidth
consumption and # OpenFlow commands
generated for different network sizes
Comparison of proposed algorithm and other
approaches in terms of bandwidth consumption
and # OpenFlow commands generated for
different homogeneity levels of requests
(small-scale network)
Request sets (RS): homog. LQ / HQ
heterogeneous

21. Player Adaptation Logic and Quality of Experience
● Bitrate adaptation schemes
Client-based, server-based, network-assisted, hybrid, ML-based
● Application/transport layer enhancements
HTTP/2 (TCP) and HTTP/3 (QUIC), proprietary formats (SRT, RIST, …), WebRTC,
low-latency/delay
● Client playback improvements
User-aware playback, content-enhancement filters, super-resolution
● Quality of Experience
Objective and subjective quality assessment, models, analytics
Content Consumption and End-to-End Aspects
21
21

22. Bitrate Adaptation Schemes
Bitrate
Adaptation
Schemes
Client-based
Adaptation
Bandwidth
-based
Buffer-
based
Mixed
adaptation
Proprietary
solutions
MDP-based
Server-based
Adaptation
Network-
assisted
Adaptation
Hybrid
Adaptation
SDN-based
Server and
network-
assisted
A. Bentaleb, B. Taani, A.C. Begen, C. Timmerer and R. Zimmermann, "A Survey on Bitrate Adaptation Schemes for
Streaming Media Over HTTP," IEEE Communications Surveys & Tutorials, vol. 21, no. 1, pp. 562-585, First Quarter 2019.
22
22
Adaptive bitrate (ABR) algorithms, adaptation logics, ...

23. Adaptive bitrate (ABR) algorithms choose the lowest-quality segments in the
startup phase; quality switches due to throughput fluctuations.
H2BR: complementary to existing ABR utilizing HTTP/2 features
● Server push: piggyback retrans. segments
● Stream priority: for concurrent streams
● Stream termination: for retrans. segments
HTTP/2-Based Retransmission (H2BR)
23
23
Minh Nguyen, Christian Timmerer, and Hermann Hellwagner. “H2BR: An
HTTP/2-based retransmission technique to improve the QoE of adaptive
video streaming”. 25th ACM Workshop on Packet Video (PV'20), June 2020.

24. Client/server architecture with two computers
● HTTP/2 server and HTTP/2 client (both based on nghttp2)
● Dummynet emulates a state-of-the-art mobile network trace
Video content
● Big Buck Bunny: 596 seconds
● Segment duration: 1s, 2s, 4s, 6s
● Quality: 20 versions
● Resolutions: 320x240, 480x360, 854x480, 1280x720, 1920x1080
Compared method
● SQUAD1)
H2BR Evaluation Setup
24
24
Dummynet
Throughput-based AGG
Buffer-based BBA
Hybrid SARA
Last throughput
1)
Cong Wang, Divyashri Bhat, Amr Rizk, Michael Zink..
“Design and Analysis of QoE-Aware Quality Adaptation
for DASH: A Spectrum-Based Approach”. ACM Trans.
Multimedia Comput. Commun. Appl. 13, 3s, Article 45,
August 2017.

25. H2BR Experimental Results
25
25
Overall QoE score based on the ITU-T P.1203 QoE model mode 0
CDF of video quality in an experimental run (segment duration = 4s)
H2BR can decrease lowest-quality
playback by up to more than 70%
QoE increase by up to 13%

26. Quality of Experience (QoE) ...
● “... is the degree of delight or annoyance of the user of an application or service.
It results from the fulfillment of his or her expectations with respect to the utility and / or
enjoyment of the application or service in the light of the user’s personality and current state.”1)
● … can be easily extended to various domains, e.g., immersive media experiences.2)
26
26
1)
P. Le Callet, S. Möller, A. Perkis, et al. QUALINET White Paper
on Definitions of Quality of Experience. European Network on
Quality of Experience in Multimedia Systems and Services
(COST Action IC 1003). 2012.
2)
A. Perkis, C. Timmerer, et al. 2020. QUALINET White Paper on
Definitions of Immersive Media Experience (IMEx).
arXiv:2007.07032 [cs.MM]
3)
Jeroen van der Hooft, Tim Wauters, Filip De Turck, Christian
Timmerer, and Hermann Hellwagner. “Towards 6DoF HTTP
Adaptive Streaming Through Point Cloud Compression”.
27th ACM Int’l. Conf. on Multimedia (MM'19). Oct. 2019.
3)

27. QoE Evaluation for Adaptive Point Cloud Streaming
Volumetric media delivery with six degrees of freedom (6DoF) experience,
but significant bandwidth consumption
27
27
Jeroen van der Hooft, Maria Torres Vega, Filip De Turck, Christian Timmerer, Raimund Schatz, and Ali C. Begen.
“Subjective and Objective QoE Evaluation for Adaptive Point Cloud Streaming”.
12th Int’l. Conf. on Multimedia Quality of Experience (QoMEX’20). May 2020.
Slides courtesy of Jeroen van der Hooft (adapted)
4.1 Gbps 3.8 Gbps 5.7 Gbps 5.6 Gbps

28. QoE Evaluation for Adaptive Point Cloud Streaming
Volumetric media scene
28
28

29. QoE Evaluation for Adaptive Point Cloud Streaming
Point cloud compression (PCC, with MPEG reference encoder)
29
29
5.7 Gbps 40.4 Mbps 4.5 Mbps

30. Research questions:
1. What is the impact of network and content characteristics on the perceived
video quality?
2. How do objective metrics correlate with
subjective ratings for perceived video quality?
Methodology:
● 3 different video sequences were created
● Each one in 8 different configurations:
● Participants were asked to rate video quality
● Objective metrics were computed
QoE Evaluation for Adaptive Point Cloud Streaming
30
30
Bandwidth [Mbps] Allocation Prediction
20 Visible objects 0
60 Visible objects 0
100 Visible objects 0
20 Visible objects 1
60 Visible objects 1
100 Visible objects 1
60 All objects 0
∞ N/A N/A

31. Sample results (for RQ 1):
QoE Evaluation for Adaptive Point Cloud Streaming
31
31
Subjects can distinguish between different
bitrates
Viewport prediction allows to improve the
observed quality

32. Sample results (for RQ 2):
QoE Evaluation for Adaptive Point Cloud Streaming
32
32
Clear correlation between objective
metrics and MOS scores
Subjective scores match best with SSIM

33. Content Delivery
● Edge computing support
● CDNs, SDN, NVF, … ⇿ clients
● Low-latency live streaming
HTTP Adaptive Streaming – Where Is It Heading?
33
33
Content Consumption & E2E
● Client-side content improvement, SR
● Machine learning
● Appl. & transport layer enhancements
https://athena.itec.aau.at/
Content Provisioning
● Content and context awareness
● Multi-codec (AVC, HEVC, VVC, VP9, AV1)
● Machine learning
Quality of Experience
● Immersive content
● User-aware playback
● Better/new QoE models and analytics

34. ATHENA team
Hadi Amirpour, Jesús Aguilar Armijo, Ekrem Çetinkaya, Alireza Erfanian, Reza Farahani,
Mohammad Ghanbari, David Langmeier, Vignesh V Menon, Minh Nguyen, Babak
Taraghi, Farzad Tashtarian; Hermann Hellwagner, Christian Timmerer
(Inter-)National Collaborators
● Prof. Ali C. Begen, Ozyegin University, Turkey
● Prof. Filip De Turck, Ghent University – imec, Belgium
● Dr. Jeroen van der Hooft, Ghent University – imec, Belgium
● Prof. Mohamed-Chaker Larabi, University of Poitiers, France
● Dr. Raimund Schatz, AIT Austrian Institute of Technology, Austria
● Prof. Roger Zimmermann, NUS, Singapore
● Dr. Abdelhak Bentaleb, NUS, Singapore
Acknowledgments
34
34
https://athena.itec.aau.at/

35. Thank you for your attention
35