4월 26일 AWSKRUG (AWS 한국 사용자 모임) 아키텍쳐 모임에서 발표한 "Serverless Media Workflow" 발표 슬라이드입니다.

1. Serverless
Media Workflow

2. Speaker
• Serverless Lover
• Software Engineer @ Vingle
• Freelancer Software Engineer
@ Artistscard
• Software Engineer @
Law&Company
• https://github.com/mooyoul
• https://mooyoul.github.io

4. A journey to
Video Service 🎥

5. Video is becoming
the most important content
🔥🔥🔥

6. Let’s add video feature
on Vingle! 😎

7. …but how?

8. Common VOD
workflow

9. Common Video workflow

10. Common Video workflow

11. Common Video workflow

12. Common Video workflow

13. Building Ingest block

14. Ingest Step Requirements
• Support File-based workflow only for now
• Simple upload workflow for clients
• Fast upload speeds anywhere in the world
• Durable, scalable, fast storage
• No additional disk storage required for upload
• No maintenance required

15. Ingest Step Requirements
• Support File-based workflow only for now
• Simple upload workflow for clients
• Fast upload speeds anywhere in the world
• Durable, scalable, fast storage
• No additional disk storage required for upload
• No maintenance required

16. S3 Upload using pre-signed
URL
• Allows client can upload
directly to S3 using pre-
signed URL
• AWS-SDK supports pre-
signed url generation

17. S3 Upload using pre-signed
URL

18. …but we are using
us-east-1 region

19. some regional ISPs have
bandwidth issue on
international network transfer 💩

20. Slow Upload on some ISPs

21. Upload speed is so
slow!🐢🐢🐢

22. Submarine cables
in South Korea

23. Submarine cables
in South Korea

24. Two options to
accelerate S3 transfer 🚀

25. Two options to accelerate S3 transfer
• Proxy file transfer traffic through CloudFront
• Cloudfront can accelerate dynamic content
• Uses Cloudfront Edge locations to optimize route path
• Enable S3 Transfer Acceleration
• Takes advantage of CloudFront’s globally distributed
edge locations
• More transfer optimizations

26. S3 Transfer Acceleration is faster than Cloudfront as
Proxy

27. S3 Transfer Acceleration makes throughput
faster

28. Ingest block architecture

29. Problem solved! ✅

30. Building Processing
block

31. Processing Requirements
• use fully-managed service if possible
• scalable, reliable
• adaptive bitrate streaming
• Important: compatible with various platforms, devices
• Desktop Web - IE 10+, Edge, Chrome, Firefox
• Mobile Web - Mobile Safari, Chrome Mobie
• Android - 4.1 ICS+
• iOS - 9+
• Most Important: Delivery video to users ASAP

32. Wait!
What is Adaptive Bitrate
Streaming?

33. Adaptive Bitrate Streaming
• From Wikipedia:
• Adaptive bitrate streaming is a technique used in
streaming multimedia over computer networks.
• It works by detecting a user's bandwidth and
CPU capacity in real time and adjusting the
quality of a video stream accordingly.
• The result: very little buffering, fast start time and a
good experience for both high-end and low-end
connections.

34. Adaptive Bitrate Streaming

35. Choosing
Streaming Protocol

36. Choosing Streaming
Protocol
• Non-HTTP based protocols like RTMP are not in our interest
• Looked for HTTP-based protocols, or methods
• Progressive Download Single File via HTTP
• 1st request: Range: 0-255
• 2st request: Range: 256-511
• 3rd request: Range: 512-767
• …
• Apple HLS
• MPEG-DASH

37. Choosing Streaming
Protocol
• Progressive Download Single File via HTTP
• Apple HLS
• MPEG-DASH

38. Choosing Streaming
Protocol
• Progressive Download Single File via HTTP
• Apple HLS - Winner!
• MPEG-DASH

39. Why we choose HLS

40. Progressive Download
Single File via HTTP
• Pros
• Easy to implement
• Cons
• Implementing Adaptive Bitrate Streaming is so hard (almost
impossible) without getting “choppy” frames, or “pop” gap
sound between streams
• Slow seeking/starting since container requires metadata reading
• Large binary can occur lower cache hit
• No Live Stream compatibility

41. MPEG-DASH
• Pros
• New standard
• Based on Fragmented MPEG4 (similar to MPEG2 Transport Stream on
HLS)
• Broader codec support
• Cons
• IMO it is not production ready
• Market, Platform, Community support is not good at this moment
• Workflow was unstable during experiment

42. First Video
Specification

43. First Spec - Protocol
• HTTP Live Stream (HLS)
• Apple invented standard, natively supported on macOS/iOS/tvOS
• Natively supports adaptive bitrate streaming, Live Streaming, and DRM
(Apple FairPlay)
• Based on HTTP Protocol, Play well with transitional HTTP CDN
• Trusted by many providers, Many stable implementations
• Amazon, Youtube, Facebook, Twitter, Vimeo, Twitch, Akamai, New
York Times, FOX TV, TED, VK, POOQ, SK Broadband TV …
• Android - ExoPlayer
• Web - MediaSource Extension is standard spec, HLS.js, Video.js

44. First Spec - Input
• No codec limitation of input source
… but expects most input source are H.264/AVC
+ AAC based
• Max acceptable resolution: 1080p
• Max acceptable bitrate: 10Mbps
• Max acceptable duration: 5 minutes

45. First Spec - Output
• Common
• H.264/AVC1 + AAC LC
• Many graphic chipset supports hardware accelerated decoding
• Most popular codec in market
• Apple HLS
• Three variants for adaptive bitrate streaming
• High Quality: Up to 1080p, 60p, High/4.1, 4Mbps
• Medium Quality: Up to 720p, 30p, Main/4.0, 2Mbps
• Low Quality: Up to 480p, 30p, Baseline/3.1 800kbps
• Progressive Download Single File via HTTP for HLS fallback
• Medium Quality only

46. First Spec - Output
• 4 Outputs
• HLS HQ
• HLS MQ
• HLS LQ
• HTTP Progressive MP4 MQ

47. Check requirements

48. Requirements
• use fully-managed service if possible
• scalable, reliable
• adaptive bitrate streaming
• Important: compatible with various platforms, devices
• Desktop Web - IE 10+, Edge, Chrome, Firefox
• Mobile Web - Mobile Safari, Chrome Mobie
• Android - 4.1 ICS+
• iOS - 9+
• Important: Delivery video to users ASAP

49. Requirements
• use fully-managed service if possible
• scalable, reliable
• adaptive bitrate streaming
• Important: compatible with various platforms, devices
• Desktop Web - IE 10+, Edge, Chrome, Firefox
• Mobile Web - Mobile Safari, Chrome Mobie
• Android - 4.1 ICS+
• iOS - 9+
• Important: Delivery video to users ASAP

50. Requirements
• use fully-managed service if possible
• scalable, reliable
• Important: Delivery video to users ASAP

51. AWS provides bunch
of media products 🎉

52. Video Encoding Products
• AWS Elastic Transcoder
• Fully-managed service, pay only for what you use (per minute of output video)
• HLS Support
• Configurable Presets
• AWS Elemental MediaConvert
• Newly launched, Fully-managed service, pay only for what you use (per minute of output video)
• HLS Support
• Broadcasting-grade, high quality output, Professional Codec Support (e.g. ProRes)
• Configurable Presets
• In some scenarios, MediaConvert is cheaper than Elastic Transcoder
• New! Basic Tier for Simple Web Videos

53. …or build your own
video encoding service

54. Most time-consuming
task is video encoding

55. Tested two products

56. Test conditions
• Input
• f(x) 4 Walls M/V (3m 33s)
• H.264/AVC, 1080p, 23.97fps, 8Mbps, High/4.1
• AAC LC, 192k, 48KHz, 2 Channels
• Mr. Robot Clip (52m 54s)
• H.264/AVC, 720p, 23.97fps, 1.5Mbps, High/4.1
• AC3, 384k, 48KHz, 6 Channels
• Output
• Single HLS Output
• H.264/AVC, 1080p, 23.97fps, ABR 3Mbps (Max 4.5Mbps), High/4.1
• AAC LC, 128k, 48KHz, 2 Channels
• No Thumbnail Generation

57. Elastic Transcoder

58. Elastic Transcoder

59. Elemental MediaConvert

60. Elemental MediaConvert

61. Results
• Overall workflow was stable
• Quality
• The output quality of both was similar
• Fixed GOP => Consistant segment duration
• Processing Speed (sorted by faster)
• Elastic Transcoder
• MediaConvert Basic
• MediaConvert Pro

62. Problems
• Pricing is quite expensive
• Needed more faster encoding speed without losing quality
• Missing CRF (Constant Rate Factor) bitrate mode support
• Can’t control encoder options
• MediaConvert has better encoder option (e.g . quantize algorithm -
CABAC)
• Can’t control video filters
(e.g. lanczos for resizing, yadif for deinterlacing, color filters like
instagram)
• There’s no way to remux video

63. Make our own
processing workflow 💥

64. Three ways to run job
• Batch - EC2
• Fargate - ECS
• Step Functions + Lambda

65. Three ways to run job
• Batch - EC2
• Fargate - ECS
• Step Functions + Lambda - WINNER!

66. Three ways to run job
• Batch - EC2
• Requires additional instance provisioning time (extremely slow)
• Fargate - ECS
• Requires additional container provisioning time
(about 30sec~60sec)
• Setting High CPU power requires High Memory Size
• Lambda
• Computing Power is quite slower than EC2/Fargate instances
• …but now we have 3008MB Memory option, which has faster processing power
• No additional provisioning time required

67. Video Processing
Workflow

68. Video Processing Workflow

69. Single Video Processing
Function

70. Issue #1

71. How to get encoding
software running on AWS
Lambda?

72. Lambda-runtime compatible
build
• Linux program
• Lambda execution runtime is based on Amazon Linux AMI (amzn-ami-hvm-2017.03.1.20170812-
x86_64-gp2)
• Instead of using build instance, just make build image which is based on amazon linux docker
image
• docker pull amazonlinux:2017.03.1.20170812
• vim Dockerfile # Be careful updating openssl. it can make incompatible shared library issue
• docker build . -t my-awesome-lambda-build-env
• docker run --rm -v $PWD:/workspace my-awesome-lambda-build-env /workspace/build.sh
• Test built binaries or libraries using lambci lambda image
• docker pull lambci/lambda:nodejs8.10
• docker run --rm -v $PWD:/workspace --entrypoint bash lambci/lambda:nodejs8.10 /
workspace/test.sh

73. Lambda-runtime compatible
build
• Node.js Native Addons
• lambci provides docker images to rebuild
native addons
• docker pull lambci/lambda:build-nodejs8.10
• docker run --rm -v $PWD:/var/task lambci/
lambda:build-nodejs8.10 # runs “npm
rebuild”

74. Issue #2

75. Timeout issues
Hard to organize each
steps

76. Step functions
• Makes workflow simple to orchestrate AWS Lambda functions
• Supports Parallel state
• built-in error handling, retry, fallback conditions
• Useful patterns
• Delay execution for X seconds or until specific date/time
• Recursive Execution using Choice State
• Scan all records from DynamoDB table
• dequeue messages from SQS until queue is empty

77. Single Video Processing
Function

78. Break logics

79. State Machine - 1st version

80. Issue #3

81. VPC makes trouble

82. VPC Issue
• Assigning Lambda to VPC introduces long cold-start time
• about 10 sec, even used 3008MB memory size
• impacts overall video processing time
• Occasionally we had “connection timeout” issues
• AWS Support confirmed sometimes ENI may not have
internet connectivity
• we had to implement our own retry logic

83. VPC Issue
• Scaling Issue when invocation spikes

84. VPC Issue
• VPC was required for S3 access through S3
Endpoint only (to authenticate s3 access using VPCE
ID)
• Removed VPC configuration
• Switched to stream data using pre-signed urls or
download object using AWS-SDK
• reduced cold-start time (<40ms)
• “connection timeout” issue has gone

85. Issue #4

86. Tiny ephemeral
storage (/tmp)

87. Tiny ephemeral storage (/
tmp)
• Ephemeral storage capacity is 512MB, too small for media processing
• You can’t add block device to lambda
• NFS / EFS
• Lambda does not provide root permission
• so you cannot mount disks - you cannot use NFS in Lambda
• Ramdisk
• Ramdisk like /dev/shm is not available on Lambda
• Lambda does not provide root permission
• so you cannot create or mount ramdisk

88. Tiny ephemeral storage (/
tmp)
• S3 - only available option for now
• stream everything if possible
• use s3 as temporary storage
• Video Input - S3 supports seeking object using Range request header => can be streamed
• Video Output - may vary depending on container
• MP4 - requires seekable output => you cannot pipe output => you cannot stream outputs
• HLS - does not support writing outputs to pipes, since HLS consists of several files
• Watch file system (e.g. chokidar)
• Upload segments to s3 if generated
• Removed uploaded file
• Note: Higher memory size option provides faster Disk I/O and Network I/O
• Note: S3 Object keys are stored in UTF-8 binary ordering across multiple partitions in the index
=> UUID key prefix helps to reduce first byte latency

89. Issue #5

90. Optimize container
reusability

91. Optimize container
reusability
• Invoking different functions
=> lambda container cannot be reused

92. Optimize container
reusability
• HACK: Group multiple lambda functions to single lambda
function
• Inject task name using Pass State

93. Processing Pipeline
Architecture - 1st version

94. Building Delivery block

95. Delivery block
• Currently we don’t need any DRM or AD
insertion solution
• Simple architecture
• Use S3 as origin, Cloudfront as CDN
• Use maximum TTL to maximize cache hit

96. Delivery block architecture

97. Connect blocks to
make pipeline

98. Media Pipeline Architecture

99. Media Pipeline Architecture

100. Media Pipeline Architecture

101. Media Pipeline Architecture

102. Launch

103. Appearing video uploads

104. Wait! There’s more!

105. 2nd Version
• Improves encoding speed, output quality
• Support Parallel Encoding
• Split video streams to small chunks
• Split Video/Audio encoding process
• Encode streams per each preset, not outputs
• HLS HQ / HLS MQ / HLS LQ / MP4 MQ
=> Encode streams to HQ / MQ / LQ and assemble them
• Bypass encoding if input is compatible with our spec
• Skip HLS encoding process if source quality is not good, or duration is too short
• Support GIF input/output

106. Parallel Encoding

107. Break logics

108. State Machine - 2nd version

109. Results
• Fast processing without losing quality
• now processing f(x) M/V (1080p, 3m 33s) just takes 60 sec
• if input source is compatible with our spec, processing can be done within <10 sec
• Great output quality
• Saving cost
• encoding is not required if input source is compatible with our spec
• Stable workflow
• There’s no resource to manage
• Acceptable max bitrate: Up to 300Mbps (Tested with ProRes source)
Acceptable max resolution: 4K
Acceptable codec: HEVC, ProRes
Acceptable max duration: 1 hour

110. Further Steps
• Support 1 hour+ video processing using Elastic Transcoder
• Support Live Streaming
• MediaLive? Fargate?
• Finding solution for WebRTC gateway to relay live video stream
• Video Advertisement
• MediaTailor?
• Metrics, Management Tools …
• NSFW filtering - rekognition video or rekognition

111. Postmortem

112. •
• Elastic Transcode Elemental MediaConvert
• Processing pipeline
VOD
• Lambda
• Lambda

113. How about other media
types?
(Audio, GIF, Image…)

114. We’re hiring!
https://careers.vingle.net

115. Thank you
* * *