Miska Hannuksela: Versatile Video Coding Standard H.266/VVC | 30 years of success – image and multimedia standardization in 2021

1. 30 years of success – image and multimedia standardization in 2021 16.2.2021 Miska M. Hannuksela, Nokia

3. © 2021 Nokia 2 Agenda Public 1. Motivation: why a new video codec 2. Versatile Video Coding standard • Development phases • Compression performance • Profiles • Application examples • Some implementations 3. Summary

4. © 2021 Nokia 3 Public Miska M. Hannuksela Presenter Head of Video Research in Nokia Technologies Nokia Bell Labs Fellow LinkedIn, Google Scholar Active delegate in video coding standardization since 1999 H.263++, H.264/AVC, H.265/HEVC, H.266/VVC In H.266/VVC Co-author of >100 input documents Key contributor in many high-level codec features Editor in the standard amendment for encapsulating H.266/VVC in .mp4 files

6. © 2021 Nokia 5 Public Internet traffic Amount of video traffic will grow from 2017 to 2022 by >4x Data source: Cisco Visual Network Index Complete Forecast Highlights 2017-2022 0 100 200 300 400 2017 2022 video 70% video 80% Global total Internet traffic / month (EB)

7. © 2021 Nokia 6 Public Mobile data traffic Amount of video traffic will grow from 2020 to 2026 by >5x 0 50 100 150 200 250 2020 2026 video 66% video 77% Global total mobile data traffic / month (EB) Data source: Ericsson Mobility Report, Nov 2020

8. © 2021 Nokia 7 Public Video coding standards H.264 AVC 2003 2013 2020 H.265 HEVC Bit rate 50% bit rate reduction 50% bit rate reduction (H.266) Advanced Video Coding (H.264/AVC) High Efficiency Video Coding (H.265/HEVC) Versatile Video Coding (H.266/VVC)

9. © 2021 Nokia 8 Public Development focus of video coding standardization H.264 Resolutions up to high definition (HD) 1920x1080 H.265 Resolutions up to ultra high definition (UHD) 3840x2160 H.266 Resolutions up to 8K UHD 7680x4320 High dynamic range Screen content 360° video H.264/AVC, H.265/HEVC, and H.266/VVC: Aim at the best compression for storing and distributing video and are suitable for widespread use … but were specifically tested and developed for the following cases:

11. © 2021 Nokia 10 Oct '17 Call for Proposals Issued jointly by ITU-T VCEG and ISO/IEC MPEG Categories for standard and high dynamic range (SDR, HDR) and 360° video 23 submissions received from 32 organizations Subjective viewing comparisons Public Development phases Versatile Video Coding standard Oct '15 Exploration phase Joint Video Exploration Team (JVET) of ITU-T VCEG and ISO/IEC MPEG was established Jointly maintained public codec software (JEM) maintained to explore new coding tools Compression performance of JEM served as an evidence to start a standardization project Apr '18 Collaborative phase JVET converted to Joint Video Experts Team https://www.jvet-experts.org/ Meetings every ~3 months, tens of organizations and hundreds of experts participating. Up to >1000 technical input documents per meeting. Jointly maintained public standard draft and codec software Aug '20 Final standard https://www.itu.int/rec/T-REC-H.266 ITU-T International Telecommunication Union, Telecommunication Standardization Sector VCEG Video Coding Experts Group ISO International Organization for Standardization IEC International Electrotechnical Commission MPEG Moving Picture Experts Group

12. © 2021 Nokia 11 Public Verification testing Compression performance Verification testing = subjective testing to verify the performance of a standard SDR UHD (Standard Dynamic Range, Ultra High Definition) results (JVET-T0097) Comparison anchor: HEVC reference SW (HM), no perceptual optimization Tested: VVC reference software (VTM), no perceptual optimization Fraunhofer HHI's VVenC, medium preset (110x faster than VTM), perceptual optimization Presenter's conclusion: the performance is close to the targeted 50% bitrate reduction, with demanding test sequences Next: SDR HD, HDR HD, HDR UHD and 360° testing VTM VVenC DrivingPOV3 -61% -63% Marathon2 -37% -42% MountainBay2 -37% -39% NeptuneFountain3 -38% -52% TallBuildings2 -41% -51% Overall -43% -49% Overall (BDBR Y-PSNR) -30% N/A Subjective Bjontegaard delta bitrate (MOS)

13. © 2021 Nokia 12 Public Profiles of the Versatile Video Coding standard Multilayer Main 10 4:4:4 Additional chroma formats and coding tools Scalability with inter-layer prediction Multilayer Main 10 Scalability with inter-layer prediction E.g. for spatial or multiview scalability Main 10 4:4:4 Additional chroma formats and coding tools Main 10 4:2:0 chroma format, ≤10 bits / color component Main 10 Still Picture Main 10 4:4:4 Still Picture profile = a subset of algorithmic features and limits that shall be supported by decoders

14. © 2021 Nokia 13 Adaptive resolution change Public Application examples of versatility 1/2 Versatile Video Coding Ultra-low end-to-end delay (e.g. for cloud gaming and video conferencing) • Reduced end-to-end delay compared to earlier codecs thanks to gradual decoding refresh Adaptive resolution change • Reference picture resampling tool enhancing compression when picture width/height changes • ~5% reduced bitrate estimated for live streaming • For video conferencing, adaptation of encoded resolution and bitrate to network throughput, transmission loss rate, and active speaker • More info: JVET-M0259 4K HD 720p (time) segments … … dirty area clean area … Gradual decoding refresh

15. © 2021 Nokia 14 Viewport-dependent 360° Example: 6K viewport resolution with 4K decoding capacity Public Application examples of versatility 2/2 Versatile Video Coding Screen content coding (e.g. for game streaming, cloud gaming, screen/slide sharing) • Specific screen content coding tools 360° virtual reality • Viewport-dependent streaming more efficient and straightforward to implement through subpictures • Coding tools inherently supporting omnidirectional projection formats Multiview 3D and spatial scalability • No additional coding tools needed, since inter-layer prediction integrated with motion compensation • Reduced complexity and memory usage for spatial scalability viewport viewport

16. © 2021 Nokia 15 Public Some future development Versatile Video Coding Apr 2021 Standard for encapsulating VVC into .mp4 files and for streaming Amendment 2 of ISO/IEC 14496-15:2015 → ISO/IEC 14496-15:2021 Jul 2021 Standard for encapsulating VVC into the High Efficiency Image File Format (HEIF) Amendment 2 of ISO/IEC 23008-12:2017 Jul 2021? VVC verification testing finalization Oct 2021? VVC software and conformance bitstream finalization Jan 2022 VVC version 2 finalization Range extensions, particularly >10 bits per color component Additional supplemental enhancement information 2021-22? Real-time transport protocol payload format for VVC; needed for video conferencing, cloud gaming, and other very-low-delay applications VVC ISO/IEC 14496-15: Carriage of network abstraction layer (NAL) unit structured video in the ISO base media file format ISO base media file format (ISOBMFF) ≈ .mp4 Streaming protocols

17. © 2021 Nokia 16 Some implementations of the Versatile Video Coding standard Summary 1. Several publicly available source code packages have been made available 2. VVenC by Fraunhofer HHI is optimized for speed and compression and is publicly available 3. Real-time software decoding has been demonstrated on smartphones and computers Details H.266/VVC software encoders for computers VVC test model (VTM), source code publicly available, jointly developed by JVET VVenC by Fraunhofer HHI, source code publicly available H.266/VVC software decoders for mobile platforms / smartphones: Ali266 (Alibaba). Real-time 4K decoding on some smartphones Tencent O266. Real-time 4K decoding on some smartphones H.266/VVC decoder by Ittiam H.266/VVC software decoders for computers VVC test model (VTM), source code publicly available, jointly developed by JVET Multi-threaded VTM decoder by InterDigital, source code publicly available VVdeC by Fraunhofer HHI, source code publicly available H.266/VVC software decoder by Sharp, reportedly capable of 8K decoding Public

18. © 2021 Nokia 17 Public Summary Versatile Video Coding (H.266/VVC) standard was finalized mid 2020 VVC provides • The same picture quality with ~50% less bitrate compared to the previous generation (H.265/HEVC) • Inherent support of many applications and services First VVC codec implementations available