My session for WebRTC Summit, Nov 2015, Santa Clara WebRTC sits at the intersection between VoIP and the Web. As such, it poses some interesting challenges for those developing services on top of it, but also for those who need to test and monitor these services. In this session, Tsahi Levent-Levi will review the various challenges posed by WebRTC when it comes to testing and monitoring and on ways to overcome them.

1. Overcoming the Challenges in
Testing WebRTC Services
Tsahi Levent-Levi
November 2015

2. Let me tell you a little secret
People don’t usually test
WebRTC implementations…

3. Why is that?
• WebRTC is still niche/experimental to their
business
• They are a startup with limited resources
• Adding features takes priority over stability
• Is this VoIP or Web?

4. At an intersection of worlds
4
Our
Internet
Web
Traditional
VoIP

5. The world of VoIP testing
• Focused on DUTs (Devices Under Test)
• Rely heavily on slow moving standards and
implementations
• Vast majority of testing solutions are on
premise
• Assume limited geographies
5

6. The world of Web testing
• Focused on different browsers
• Rely heavily on the GUI
• Responsive design causes GUI changes
• GUI and API testing treated as separate
• Geography doesn’t matter for functionality
6

7. 5 Major Challenges
When trying to test your WebRTC service

8. Browser version
changes
Do you know if your service will not break next
week?
#1

9. Browsers update cycle in 2015
This translates to a regression test needed on a monthly
basis – not taking into account changes you make to the
service
40
41
42
43
44
45
46
35
36
37
38
39
40
Dec-14 Feb-15 Apr-15 May-15 Jul-15 Sep-15 Oct-15 Dec-15
Chrome
Firefox

10. Stable, Beta, Dev and Canary
Stable What your customers are using
Beta What you should expect in the next release
Dev
Where you can complain about breakage – and expect fixes
in time
Canary
A nightly build of whatever is available at that moment in
time
Testing only against the Stable version of browsers means you find out
issues only after the service breaks for your customers.

11. BrowserYour app
A Word about Mobile
At what pace do you update you app’s WebRTC lib?
WebRTC WebRTC
6-8 weeks update cycle

12. NAT traversal
How does your service fair in varying network
conditions?
#2

13. WebRTC isn’t always P2P
Anywhere between 5-50% of your calls get
relayed via TURN servers
client client
Web
server
TURN
server

14. Have you setup your relays
properly?
• Support for UDP
• Support for TCP

15. Location, Location, Location
When a call in Paris gets relayed through a TURN server
in the US – you lose a customer…

16. Server scale
Have you tested your service with 10 browsers?
100? 1,000? More?
#3

17. What do you test at scale?
• Signaling?
• Media relays?
• Backend media processing?
• Single session multiparty scale?
• End-to-end service scale?
• Do you add variance to your scale testing?

18. “
A typical conversation…
Oh, it should work for 100 people in the
same conference. We did test it with 12
people – the number of machines we had
available – and it worked perfectly fine.

19. Service uptime
Do you know if your service is really running?
#4

20. What people monitor today
Memory
Storage
CPU
Network
Ping

21. End-to-end
• There are many moving parts in your service
• WebRTC requires 3 or more servers to operate
• You need to make sure it runs end-to-end

22. Orchestration
How do you automate the mechanics of
communications in your service?
#5

23. Web testing is simpler (most of
the time)
1. Write your script for a browser
2. Run it once
3. Run it in parallel on 10,000 browsers at once
4. You’re done

24. WebRTC is about coordinating
people
1. A user and an agent
2. A user and a PSTN/mobile call
3. N users joining in on a single call
4. Automatic/dynamic/manual pairing of the
users
5. …

25. Don’t take WebRTC
testing lightly

26. About testRTC
Come check us out at testRTC.com

27. Thank You!
Tsahi Levent-Levi
tsahil@testRTC.com
@tsahil