Practical study of how to make AI/ML media processing leveraging WebRTC technology. In this slide, I explained how to bind WebRTC GW and media processing with serverless container architecture. I also explained Pros & Cons of this architecture for applying media. This slide was for RTC conference Korea 2018 https://rtckorea.com/ .

1. Copyright © NTT Communications Corporation. All rights reserved.
RTC Conference Korea 2018 on November 1st, 2018
Kensaku Komatsu
NTT Communications
Media Processing with
Serverless Architecture

2. Copyright © NTT Communications Corporation. All rights reserved.
Kensaku Komatsu
Technical manager of ECL SkyWay
Web application evangelist
at NTT Communications
in Silicon valley office
@komasshu
kensakukomatsu

3. Copyright © NTT Communications Corporation. All rights reserved.
Main Idea of the tech talk today
3
A practice of Serverless Architecture
for Media Processing using WebRTC.

4. Copyright © NTT Communications Corporation. All rights reserved.
AI / IoT

5. Copyright © NTT Communications Corporation. All rights reserved.
Agenda of the talk
5
• Current business in WebRTC
• WebRTC GW – enabler for AI/IoT
• Media processing with Serverless architecture

6. Copyright © NTT Communications Corporation. All rights reserved.
Agenda of the talk
6
• Current business in WebRTC
• WebRTC GW – enabler for AI/IoT
• Media processing with Serverless architecture

7. Copyright © NTT Communications Corporation. All rights reserved.
ECL SkyWay in nutshell
7
https://webrtc.ecl.ntt.com/en/

8. Copyright © NTT Communications Corporation. All rights reserved.
Service model
8
P2P / SFU
connectivity
Client SDK

9. Copyright © NTT Communications Corporation. All rights reserved.
Customer journey : rarejob ( online education )
9https://japan.cnet.com/article/35116616/

10. Copyright © NTT Communications Corporation. All rights reserved.
Use cases
10
• Online Education
• Online healthcare
• Field engineer support
• …

11. Copyright © NTT Communications Corporation. All rights reserved.
Wish from our customer
11
• Recording
• Voice recognition, translation
• Video detection, recognition
• Emotion detection
• Low latency live streaming
• VR / AR / Avatar
• …

12. Copyright © NTT Communications Corporation. All rights reserved.
• Recording
• Voice recognition, translation
• Video detection, recognition
• Emotion detection
• Avatar
• …
Question
12
How do we support these needs
from our customer?

13. Copyright © NTT Communications Corporation. All rights reserved.
Agenda of the talk
13
• Current business in WebRTC
• WebRTC GW – enabler for AI/IoT
• Media processing with Serverless architecture

14. Copyright © NTT Communications Corporation. All rights reserved.
Our answer
14
Use WebRTC GW
for server side media processing

15. Copyright © NTT Communications Corporation. All rights reserved.
What’s SkyWay WebRTC GW
15
• Protocol gateway of WebRTC
• Enabler for server side media processing
• Can get RTP media stream from client
• Controllable with REST API
• Easy to connect from skyway client
• Supported platform : Linux and Windows
WebRTC
GW
Processing
engineRTP

16. Copyright © NTT Communications Corporation. All rights reserved. 16
https://github.com/skyway/skyway-webrtc-gateway

17. Copyright © NTT Communications Corporation. All rights reserved.
Use case 1 : server side recording
17
• WebRTC GW … stream out media data as RTP
streaming
• Bind decoder / encoder when needed
• POST media data to cloud storage or else
WebRTC
GW
Decoder
/ EncoderRTP
Storage
POST

18. Copyright © NTT Communications Corporation. All rights reserved.
Use case 2 : voice recognition
18
• WebRTC GW … stream out media data as RTP
streaming
• Bind decoder to get raw PCM data stream
• POST voice data to Voice recognition PaaS
• POST recognition result to cloud database.
WebRTC
GW
Decoder
RTP
DB
POST
Voice
recognition

19. Copyright © NTT Communications Corporation. All rights reserved.
• WebRTC GW … stream out media data as RTP
streaming
• Bind decoder to get raw PCM data stream
• POST voice data to Voice recognition PaaS
• POST recognition result to cloud database.
Question
19
How do we manage these service
for each scenario?

20. Copyright © NTT Communications Corporation. All rights reserved.
Agenda of the talk
20
• Current business in WebRTC
• WebRTC GW – enabler for AI/IoT
• Media processing with Serverless architecture

21. Copyright © NTT Communications Corporation. All rights reserved.
Base idea to manage server side processing
21
• Apply micro-service model for each media
processing scenario.
• Agility, Flexibility and scalability
21
WebRTC
GW
Micro
Service
#1
RTP
Micro
Service #2
Micro
Service #3

22. Copyright © NTT Communications Corporation. All rights reserved.
Issues, we faced with
22
• We found that…
• Sometime, media processing engine makes
state issue.
• Scalability issue for each components
• Especially media encoder and recognition
process requires high resource usage.
• Have to allocate much resources for above
micro servers.

23. Copyright © NTT Communications Corporation. All rights reserved.
Serverless architecture
23
• AKA ”Function as a service”
• Event base execution model
• Each function will be executed and killed in
each session cycle
• Easy to get scalability
• No-need to care about long-term state issue
23
spawn

24. Copyright © NTT Communications Corporation. All rights reserved.
Our challenge
24
• Apply serverless architecture into server side
media processing scenario.
• Build serverless system from scratch on top of
docker container
• Current FaaS is focused to only RPC
GW Decoder recognition
GW Decoder recognition
API GW orchestrator
spawn

25. Copyright © NTT Communications Corporation. All rights reserved.
Demo
25

26. Copyright © NTT Communications Corporation. All rights reserved.
Pros and Cons
26
• Pros
• Flexibility, Scalability and Agility
• Do not need to care about execution time
than short lived functions
• Easy to re-use each components
• Each scenario is well isolated, e.g. security
perspective
• Cons
• Need to have consolidate monitoring
• Controlling each components while set
upping orchestration is difficult
• Several timing issue will happen, especially
video streaming.

27. Copyright © NTT Communications Corporation. All rights reserved.
Summary
27
• Current business in WebRTC
• It is used for bunch of use cases
• Media processing is needed
• WebRTC GW – enabler for AI/IoT
• Protocol GW for WebRTC
• Can be orchestrated with media processing
engines
• Media processing with Serverless architecture
• Flexibility, scalability and agility
• Well isolated and security
• Need to have more practice and
management system.

28. Copyright © NTT Communications Corporation. All rights reserved.
Thanks!!
@komasshu
kensakukomatsu