This session, given at Kranky Geek India, offers suggestions on how to go about building test automation that is suitable for WebRTC services.

1. Quality Assurance for WebRTC
Services
March 2016, Kranky Geek India
Tsahi Levent-Levi

2. People don’t usually test their
WebRTC implementations
Seriously
2

3. And when they do, they test
things manually
3

4. Testing and automation are
mandatory and widely used
Meanwhile, on other software fields…
4

5. How do you go about
automating your WebRTC
testbed?
7 suggestions for you
5
W. Heath Robinson

6. • WebRTC changes… monthly
• You have more than a voice call
to test
Rely on the browser
6
40
41
42
43
44
45
46
47
48
49
35
36
37
38
39
40
41
42
43
44
45
Dec-14 Apr-15 Jul-15 Oct-15 Jan-16 May-16
Chrome
Firefox
Selenium is your friend
1

7. Rely on the browser… and peer into the future
7
1
Stable What your customers are using
Beta What you should expect in the next release
Dev
Where you can complain about breakage – and expect fixes in
time
Canary A nightly build of whatever is available at that moment in time
Testing only against the Stable version of browsers means you find out issues
only after the service breaks for your customers.

8. > ffmpeg -i MYFILE.mp4 -pix_fmt yuv420p
> sed -i '0,/C420mpeg2/s//C420/' *.y4m
> google-chrome --use-fake-device-for-media-stream
--use-file-for-fake-video-capture=MYFILE.y4m
See http://testrtc.com/y4m-video-chrome/ for more details
Handle your media feeds
8
and think about the type of content you need
2

9. For some arcane reason (called AGC or something), browsers
will loop your fake video but won’t look your fake audio
Loop an input 6 times (=expand 5 minutes to 30)
> ffmpeg -f concat -i <(for i in {1..6}; do echo "file '$PWD/INPUT.wav'"; done) –c
copy OUTPUT.wav
> google-chrome --use-fake-device-for-media-stream
--use-file-for-fake-audio-capture=OUTPUT.wav
9
2 Handle your media feeds
and make sure your audio recording is suitably long

10. Take care of the network
10
Use netfilter.org iptables
Network Impairments
• Bitrate limitation
• Packet loss
• Jitter & latency
Firewall Configuration
• None
• Force TURN relay
• HTTP only
• HTTPS only
Multiple Locations
• Virtualization sucks
• Stability & Throughput
is what you need
• Explore barebone
options
3

11. • Caller / Callee
• Caller / Agent
• Caller / Expert
• Meeting point
• Conference
• Webinar
Handle synchronization
11
4

12. Can you see the problem?
Visualize the results
12
5

13. Visualize the results
13
Chrome 48 with VP8 Chrome 49 with VP9
5

14. Are these real network issues with packet losses or bugs in these
products and how they report packet loss?
Visualize the results (2)
14
5

15. • How much time does it take to connect the session?
• Are the media channels you need really there?
• Are you sure there aren’t any “additional” media channels?
• Is the bitrate following your restrictions?
Don’t forget to set your expectations
15
6

16. You never know when things will become handy…
Here are a few thoughts:
• chrome://webrtc-internals dump
• Browser console logs
• Media recordings
• Machine performance
Collect EVERYTHING
16
7

17. Don’t take WebRTC
testing lightly
17

18. Thank You
Tsahi Levent-Levi
http://bloggeek.me
tsahil@bloggeek.me
18