HLS Optimal Bitrate

動画アプリをなめらかに動かす技術
Yuji Hato
iOSDC 2018

Yuji Hato
CyberAgent, Inc. / AbemaTV, Inc.
dekatotoro
@dekatotoro
Contributed services
About me

動画再生
HLS
実装ポイント
最適なbitrate
プレイヤーの監視
Agenda

Download
Progressive Download
Streaming
動画再生
HTTP Adaptive Streaming
Local File
動画コンテンツ

Download
Progressive Download
Streaming
HTTP Adaptive Streaming
Local File
動画コンテンツ
動画再生

HDS (HTTP Dynamic Streaming) … Adobe
SS (Smooth Streaming) … Microsoft
MPEG-DASH(Dynamic Adaptive Streaming over HTTP) …
ISO/IEC 23009
HLS (HTTP Live Streaming) … Apple
HTTP Adaptive Streaming プロトコル
動画再生

Container Format / codec
Container Format Video codec Audio codec
MPEG2-TS .ts/m2t/.m2ts H.264/MPEG-2 AAC/AC-3/MP3
MP4 .mp4/.m4a H.264/Xvid/Divx//MPEG-4/H.265 AAC/MP3/AC-3/Voribis
MOV .mov/.qt H.264/MJEG/MPEG-4 AAC/MP3/LPCM
AVI .avi H.264/Xvid/Divx/MPEG-4 AAC/MP3/LPCM
WebM .webm VP8/VP9 Vorbis
etc
動画再生

Container Format / codec
Container Format Video codec Audio codec
MPEG2-TS .ts/m2t/.m2ts H.264/MPEG-2 AAC/AC-3/MP3
MP4 .mp4/.m4a H.264/Xvid/Divx//MPEG-4/H.265/AV1 AAC/MP3/AC-3/Voribis
MOV .mov/.qt H.264/MJEG/MPEG-4 AAC/MP3/LPCM
AVI .avi H.264/Xvid/Divx/MPEG-4 AAC/MP3/LPCM
WebM .webm VP8/VP9/AV1 Vorbis
etc
動画再生

Encoder Server System CDN
Live
動画再生

VOD
Encoder Server System CDN動画ソース
動画再生

Media Playlist
HLS
Master Playlist
Playlist

#EXTM3U
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=232370,CODECS="mp4a.40.2, avc1.4d4015"
gear1/prog_index.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=649879,CODECS="mp4a.40.2, avc1.4d401e"
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1927833,CODECS="mp4a.40.2, avc1.4d401f"
…
HLS
https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_4x3/bipbop_4x3_variant.m3u8
Master Playlist (.m3u8)

HLS
Header
playlist1.m3u8
playlist2.m3u8
Playlist3.m3u8
Playlist4.m3u8
BANDWIDTH=232370
BANDWIDTH=649879
BANDWIDTH=991714
BANDWIDTH=1927833
CODECS=xxxx
CODECS=xxxx
CODECS=xxxx
CODECS=xxxx

#EXTM3U
#EXT-X-TARGETDURATION:10
#EXT-X-VERSION:3
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-PLAYLIST-TYPE:VOD
#EXTINF:9.97667,
fileSequence0.ts
#EXTINF:9.97667,
fileSequence1.ts
#EXTINF:9.97667,
fileSequence2.ts
#EXTINF:9.97667,
fileSequence3.ts
#EXTINF:9.97667,
fileSequence4.ts
…
#EXT-X-ENDLIST
HLS
http://devimages.apple.com/iphone/samples/bipbop/gear1/prog_index.m3u8
Media Playlist (.m3u8)

#EXTM3U
#EXT-X-TARGETDURATION:10
#EXT-X-VERSION:3
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-PLAYLIST-TYPE:VOD
#EXTINF:9.97667,
fileSequence0.ts
#EXTINF:9.97667,
fileSequence1.ts
#EXTINF:9.97667,
fileSequence2.ts
#EXTINF:9.97667,
fileSequence3.ts
#EXTINF:9.97667,
fileSequence4.ts
…
#EXT-X-ENDLIST
ts
セグメントファイル (.ts)
HLS
http://devimages.apple.com/iphone/samples/bipbop/gear1/prog_index.m3u8
ts
ts
ts
ts

HLS
Header
segment1.ts
segment2.ts
segment3.ts
segment4.ts
length=10sec
Footer
length=10sec
length=10sec
length=10sec

HLS
https://developer.apple.com/streaming/
Documents

実装ポイント
動画再生
UIWebView / WKWebView WKWebVIewはiOS8~
MPMoviePlayerController iOS2 ~ iOS9
AVPlayer iOS4~
AVPlayerViewController iOS8~
Other Metalなど

実装ポイント
動画再生コンポーネント
AVPlayer
AVAsset
AVPlayerViewController
iOS/tvOS
AVPlayerView
macOS
AVPlayerLayer
AVPlayerItem

AVAsset
実装ポイント
動画再生の流れ
AVPlayerItem
URL
AVPlayer
AVPlayerLayer UIView

実装ポイント
動画再生
let asset = AVAsset(url: url)
let playerItem = AVPlayerItem(asset: asset)
let player = AVPlayer(playerItem: playerItem)
playerView.player = player
動画

実装ポイント
最適化する
動画再生

実装ポイント
動画
動画再生
動画
動画動画動画
動画

実装ポイント
動画A
動画再生

実装ポイント
動画B
thumbnail
動画再生
動画A
動画A破棄

実装ポイント
動画B
thumbnail
動画再生
動画A
thumbnail
動画A破棄

実装ポイント
動画B
thumbnail
動画再生
動画A
thumbnail
動画B生成

実装ポイント
動画B
動画再生
動画A
thumbnail
動画B生成

実装ポイント
動画B
動画再生

実装ポイント
実装
動画再生

実装ポイント
Playback start process
https://developer.apple.com/videos/play/wwdc2018/502/

実装ポイント
VideoPlayer class
class VideoPlayer {
private(set) var player: AVPlayer!
let playerItem: AVPlayerItem
init(url: URL) {
let asset: AVAsset = AVAsset(url: url)
playerItem = AVPlayerItem(asset: asset)
}
…
}

実装ポイント
サブスレッドでPlayerを生成
class VideoPlayer {
init(url: URL) {
}
…
}
// サブスレッドでプレイヤーを生成
streamURL.asObservable()
.observeOn(scheduler)
.subscribe(onNext: { [weak self] streamURL in
self?.createPlayer(url: streamURL) // createPlayerでVideoPlayerを生成
})
.disposed(by: disposeBag)

AVAsset
実装ポイント
サブスレッドでAVAssetとAVPlayerItemを生成
AVPlayerItem
URL
AVPlayer
サブスレッド

実装ポイント
RxでAVAssetのloadValuesAsynchronouslyをobserve
extension Reactive where Base: AVAsset {
var isPlayable: Observable<Bool> {
return Observable.create { [weak base] observer in
base?.loadValuesAsynchronously(forKeys: [#keyPath(AVAsset.isPlayable)]) {
if let me = base {
observer.onNext(me.isPlayable)
}
}
return Disposables.create()
}
}
…
}

実装ポイント
RxでAVPlayerItemのKVO, notificationなどをobserve
extension Reactive where Base: AVPlayerItem {
var status: Observable<AVPlayerItemStatus> {
return observe(AVPlayerItemStatus.self, #keyPath(AVPlayerItem.status)).filterNil()
}
var seekableTimeRanges: Observable<[NSValue]> {
return observe([NSValue].self, #keyPath(AVPlayerItem.seekableTimeRanges)).filterNil()
}
func asEndTimeObervable(notification: NotificationCenter = .default) -> Observable<Void> {
return notification.rx.notification(.AVPlayerItemDidPlayToEndTime, object: base).map(void)
}
…
}

実装ポイント
class VideoPlayer {
…
init(url: URL) {
asset.rx.isPlayable
.filter { $0 }
.observeOn(ConcurrentMainScheduler.instance)
.subscribe(onNext: { [weak self] _ in
guard let me = self else { return }
// メインスレッドで生成
me.player = AVPlayer(playerItem: playerItem)
})
}
…
}
AVAssetのplayableのtrueを待ってAVPlayerを生成

実装ポイント
AVPlayerを生成後、Playerの状態をobserve
// サブスレッド
asset.rx.isPlayable
.filter { $0 }
// メインスレッド
me.observePlayer()
me.playerCreated.accept(())
})

AVAsset
実装ポイント
メインスレッドでAVPlayerを生成
AVPlayerItem
URL
AVPlayer
サブスレッド
メインスレッ
ド

実装ポイント
AVPlayer生成処理など全てサブスレッドは？
asset.rx.isPlayable
.filter { $0 }
me.observePlayer()
me.playerCreated.accept(())
})
注: AVPlayerのKVOはmain thread推奨

実装ポイント
playerの状態をobserve
private func observePlayer() {
player.rx.periodicTime(for: CMTime(seconds: 1.0, preferredTimescale: Int32(NSEC_PER_SEC)))
.bind(to: periodicTime)
playerItem.rx.errorStatus
.bind(to: playerItemError)
playerItem.rx.asEndTimeObervable()
.bind(to: playerItemEndTime)
…
}

実装ポイント
AVPlayerItem.statusがreadyToPlayになるを待って再生開始
private func observePlayer() {
…
Observable.combineLatest(asset.rx.isPlayable,
playerItem.rx.status,
playerCreated)
.map { $0.0 && $0.1 == .readyToPlay }
.bind(to: isPlayableAndReadyToPlay)
…
}

AVAsset
実装ポイント
AVPlayerLayer
AVPlayerItem
URL
AVPlayer

実装ポイント
RxでAVPlayerLayerのreadyForDisplayをobserve
extension Reactive where Base: AVPlayerLayer {
var ready: Observable<Void> {
return observe(Bool.self, #keyPath(AVPlayerLayer.isReadyForDisplay), options: .init(rawValue: 0), retainSelf:
false)
.filter { _ in self.base.isReadyForDisplay }
.map(void)
}
…
}

実装ポイント
AVPlayerLayerのreadyForDisplayを待って表示
playerLayer.isHidden = true
playerLayer.rx.ready
self?.playerLayer.isHidden = false
})
.disposed(by: rx.disposeBag)

実装ポイント
AVPlayerとAVPlayerLayerはサブスレッドで破棄
deinit {
let layer = playerLayer
DispatchQueue.global(qos: .background).async {
layer.videoPlayer?.dispose()
layer.videoPlayer = nil
layer.player = nil
}
}

実装ポイント
動画B
thumbnail
動画再生
動画A
動画A破棄動画B生成
なめらかに！

実装ポイント
UI/UX × 動画再生
の最適なバランスを模索する
動画再生

最適なbitrate
preferredPeakBitRate
AVPlayerItem.preferredPeakBitRate

#EXTM3U
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=232370,CODECS="mp4a.40.2, avc1.4d4015"
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1927833,CODECS="mp4a.40.2, avc1.4d401f"
…
https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_4x3/bipbop_4x3_variant.m3u8
最適なbitrate

最適なbitrate
動画サイズに応じた最適化
縦横
iPhone

最適なbitrate
動画サイズに応じた最適化
iPad
縦横

最適なbitrate
端末解像度応じた最適化
https://www.apple.com/jp/ios/ios-11/

最適なbitrate
preferredPeakBitRate
端末解像度 × 動画サイズ
による最適なbitrateを指定する

Playerの監視
HLS Playback Session

AVPlayerItemErrorLog
Playerの監視
AVPlayerItemAccessLog
Playerの情報
AVPlayerItem
Notification
AVPlayer

Playerの監視
Notification
public static let AVPlayerItemTimeJumped: NSNotification.Name
public static let AVPlayerItemDidPlayToEndTime: NSNotification.Name
public static let AVPlayerItemFailedToPlayToEndTime: NSNotification.Name
public static let AVPlayerItemPlaybackStalled: NSNotification.Name
public static let AVPlayerItemNewAccessLogEntry: NSNotification.Name
public static let AVPlayerItemNewErrorLogEntry: NSNotification.Name

Playerの監視
Notification
var failedToPlay: Observable<Void> {
return NotificationCenter.default.rx.notification(.AVPlayerItemFailedToPlayToEndTime, object: base)
.map(void)
}
var stalled: Observable<Void> {
return NotificationCenter.default.rx.notification(.AVPlayerItemPlaybackStalled, object: base)
.map(void)
}
…
}

Playerの監視
Notification
extension Notification.Name {
static let timebaseEffectiveRateChanged = NSNotification.Name(rawValue:
String(kCMTimebaseNotification_EffectiveRateChanged))
}
var timebaseEffectiveRateChanged: Observable<Float64> {
return NotificationCenter.default.rx.notification(.timebaseEffectiveRateChanged, object: base.timebase)
.flatMap { [weak base] _ -> Observable<Float64> in
guard let timebase = base?.timebase else { return .empty() }
return .just(CMTimebaseGetRate(timebase))
}
}

AVPlayerItem - AVPlayerItemStatus
public enum AVPlayerItemStatus : Int {
case unknown
case readyToPlay
case failed
}
Playerの監視

AVPlayerItem - AVPlayerItemStatus
var status: Observable<AVPlayerItemStatus> {
return observe(AVPlayerItemStatus.self, #keyPath(AVPlayerItem.status)).filterNil()
}
}
playerItem.rx.status,
.filter { $0 == AVPlayerItemStatus.failed }
.subscribe(onNext: { [weak playerItem] _ in
if let event = playerItem?.errorLog()?.events.last {
print("(event.errorStatusCode): (event.errorDomain): (event.errorComment ?? "-")")
}
}
Playerの監視

AVPlayerItem - playback buffer
isPlaybackLikelyToKeepUp
isPlaybackBufferFull
isPlaybackBufferEmpty
var isPlaybackLikelyToKeepUp: Observable<Bool> {
return observe(Bool.self, #keyPath(AVPlayerItem.isPlaybackLikelyToKeepUp)
).filterNil()
}
var isPlaybackBufferFull: Observable<Bool> {
return observe(Bool.self, #keyPath(AVPlayerItem.isPlaybackBufferFull)).filterNil()
}
var isPlaybackBufferEmpty: Observable<Bool> {
return observe(Bool.self, #keyPath(AVPlayerItem.isPlaybackBufferEmpty)).filterNil()
}
…
}
Playerの監視

AVPlayerItem - playback buffer
var loadedTimeRanges: Observable<[NSValue]> {
return observe([NSValue].self, #keyPath(AVPlayerItem.loadedTimeRanges)).filterNil()
}
}
playerItem.rx.loadedTimeRanges
.subscribe(onNext: { [weak playerItem] loadedTimeRanges in
guard let playerItem = playerItem else { return }
let buffer: Double = (loadedTimeRanges.last as? CMTimeRange)
.map { range in (range.end - playerItem.currentTime()).seconds } ?? 0
print("buffer: (buffer)")
})
Playerの監視

AVPlayer - TimeControlStatus
@available(iOS 10.0, *)
public enum AVPlayerTimeControlStatus : Int {
case paused
case waitingToPlayAtSpecifiedRate
case playing
}
Playerの監視

AVPlayer - TimeControlStatus
extension Reactive where Base: AVPlayer {
var timeControlStatus: Observable<AVPlayerTimeControlStatus> {
return base.rx.observe(AVPlayerTimeControlStatus.self, #keyPath(AVPlayer.timeControlStatus))
.filterNil()
}
}
player.rx.timeControlStatus
.subscribe(onNext: { timeControlStatus in
// timeControlStatus
})
Playerの監視

Playerの監視

playerItem.accessLog()?.events
Playerの監視

if let accessLog = videoPlayer.playerItem.accessLog()?.events.last {
let log = """
uri: (accessLog.uri ?? "-")")
serverAddress: (accessLog.serverAddress ?? "-")")
mediaRequestsWWAN: (accessLog.mediaRequestsWWAN)")
numberOfMediaRequests: (accessLog.numberOfMediaRequests)")
playbackStartDate: (accessLog.playbackStartDate ?? Date.distantPast)")
playbackSessionID: (accessLog.playbackSessionID ?? "-")")
playbackStartOffset: (accessLog.playbackStartOffset)")
playbackType: (accessLog.playbackType ?? "-")")
startupTime: (accessLog.startupTime)")
durationWatched: (accessLog.durationWatched)")
numberOfDroppedVideoFrames: (accessLog.numberOfDroppedVideoFrames)")
numberOfStalls: (accessLog.numberOfStalls)")
observedMaxBitrate: (accessLog.observedMaxBitrate)")
observedMinBitrate: (accessLog.observedMinBitrate)")
switchBitrate: (accessLog.switchBitrate)”)
…
"""
print(log)
}
Playerの監視

Playerの監視

playerItem.errorLog()?.events
Playerの監視

if let errorLog = playerItem.errorLog()?.events.last {
let log = """
uri: (errorLog.uri ?? "")")
date: (errorLog.date ?? Date.distantPast)")
serverAddress: (errorLog.serverAddress ?? "-")")
playbackSessionID: (errorLog.playbackSessionID ?? “-")")
errorStatusCode: (errorLog.errorStatusCode)")
errorDomain: (errorLog.errorDomain)”)
errorComment: (errorLog.errorComment ?? "-")")
"""
print(log)
}
Playerの監視

Metrics
Playerの監視

Metrics Solution
MUX
CONVIVA
Akamai Media Analytics
YOUBORA
etc..
Playerの監視

まとめ
動画配信の技術を知る
最適な動画再生 × UXを追求する
ユーザーの視聴体験を追求する

HLS Optimal Bitrate

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to HLS Optimal Bitrate

Similar to HLS Optimal Bitrate (20)

More from Yuji Hato

More from Yuji Hato (9)

Recently uploaded

Recently uploaded (20)

HLS Optimal Bitrate

Editor's Notes