Run faster than fast

1. iOS Parallel Automation:
Run faster than fast
Viktar Karanevich · 26 May

2. I. History of iOS Automation Frameworks (10 min)
II. Path to the current approach at Badoo (10 min)
III. Summary (5 min)
IV. Movie: Test Run (3 min)

3. self.introduce =>
{
name: "Viktar Karanevich",
title: ["SSATE", "SSDET", "SQAAE"],
languages: ["Java", "Ruby"],
experience: "iOS Automation 7+ years"
}

4. BIGGER THAN DATING
348M+ registered users
50M+ downloads on iOS
800+ automated tests on iOS

5. Steve Jobs
Steve Wozniak
I. History

6. Jun 2010
UI Automation
Apple

7. Jun 2010
UI Automation
Sep 2014
UI Automation
xcrun simctl
Apple
"xcrun simctl" is introduced in xCode 6
It is command line utility to control the Simulator
(create, delete, erase, install, uninstall, ...).

8. Jun 2010
UI Automation
Sep 2015
UI Automation
xcrun simctl
UI Testing
Apple
UI Testing is introduced as replacement
for UIAutomation in xCode 7
Sep 2014
UI Automation
xcrun simctl

9. Jun 2010
UI Automation
Sep 2016
UI Testing
xcrun simctl
Apple
Sep 2015
UI Automation
xcrun simctl
UI Testing
Sep 2014
UI Automation
xcrun simctl
UI Automation removed in xCode 8

11. Jun 2010
UI Automation
Sep 2014
xcrun simctl
Sep 2016
UI Testing
Sep 2016
UI Testing
Apple
Calabash
Nov 2009
UI Spec

12. Calabash
Nov 2009
UI Spec
Feb 2011
Frank

13. Calabash
Nov 2009
UI Spec
Feb 2011
Frank
Jan 2012
Calabash
Step Definitions
Ruby Client Library
Features
Developer host
Calabash server
Simulator
Your app
UIQuery
Touch Actions

14. Calabash
Apr 2013
Calabash +
UIAutomation
Link: https://github.com/calabash/run_loop
Step Definitions
Ruby Client Library
Features
Developer host
Calabash server
Simulator
Your app
UIQuery
Run loop:
UIAutomation

15. Calabash
Sep 2016
Calabash +
DeviceAgent
Link: https://github.com/calabash/run_loop/
tree/develop/lib/run_loop/device_agent
Developer host Simulator
Step Definitions
Ruby Client Library
XCTest Runner
Features
Calabash server
Your app
UIQuery
Device Agent

16. Calabash
Nov 2009
UI Spec
Feb 2011
Frank
Jan 2012
Calabash
iOSAutomation Frameworks
Apr 2013
Calabash +
UIAutomation
Sep 2016
Calabash +
DeviceAgent
Jun 2010
UI Automation
Sep 2016
UI Testing
xcrun simctl
Sep 2015
UI Automation
xcrun simctl
UI Testing
Sep 2014
UI Automation
xcrun simctl
Apple
3 years 1 year

17. Calabash
Nov 2009
UI Spec
Feb 2011
Frank
Jan 2012
Calabash
iOSAutomation Frameworks
Apr 2013
Calabash +
UIAutomation
Sep 2016
Calabash +
DeviceAgent
Jun 2010
UI Automation
Sep 2016
UI Testing
xcrun simctl
Sep 2015
UI Automation
xcrun simctl
UI Testing
Sep 2014
UI Automation
xcrun simctl
Apple
3 years 1 year

18. II. iOS Automation at Badoo
Approach
~80
flaky tests
80
data
fail tests

19. II. Before Parallel Run
Test Suite Run
Nightly Cucumber Report
Nightmare!
15h
total time
1
simulator
80
data
~500
tests

20. II. Before Parallel Run
Test Suite Run
Nightly Cucumber Report
Nightmare!
80
data
fail tests
1simulator
Key Issue: How to speed up Test Run?
15h
total time
~500
tests
1
simulator

21. Let's Parallel!

22. Let's Parallel!
Blocker Issue:
- Calabash does not support Parallel run on 1 desktop

23. Let's Parallel
with multi-desktops!

24. https://techblog.badoo.com
Tim's talk at Badoo: "Parallel Calabash Testing on iOS"

25. II. Test Result: multi-desktops
~4 hours regression suite run time
5 Desktops

26. II. Test Result: multi-desktops
Mac PRO (24 cores)
~2 hours regression suite run time
10 Desktops

28. X Desktops: host limit reached

29. II. Clustering Hosts

30. II. Clustering Hosts
- Parallel run on multi (virtual) hosts
- Mac Pro and hosts: test suite - ~1h 30 min

31. II. Clustering Hosts
Issues:
- multi-desktops start up is Flaky
- simulators can get stuck

32. II. Clustering Hosts
Is it possible to automate multi-simulators on one desktop?

33. II. Facebook: Web Driver Agent

34. Developer host Simulator
Selenium Driver
WDA Runner
Your app
FB UIA WDA
Key Change: Reimplemented UIAutomation dependency
Link: https://github.com/facebook/WebDriverAgent

35. Profit:
- 7 simulators on single desktop
- More stable! more efficient!

36. II. Full Migration to FB UIA WDA
Issues:
- all tests have calabash code

37. II. Full Migration to FB UIA WDA
Issues:
- all tests have calabash code
- custom app backdoors

38. II. Full Migration to FB UIA WDA
Issues:
- all tests have calabash code
- custom app backdoors
- workarounds and other

39. II. Full Migration to FB UIA WDA
Full Migration Costs: ~x hours
Is it possible to do Fast Migration?

40. II. Calabash + Facebook
Fast migration: Facebash!
Step Definitions
Ruby Client Library
Features
Developer host
Calabash server
Simulator
Your app
UIQuery
Run loop:
UIAutomation

41. II. Calabash + Facebook
Fast migration: Facebash!
Step Definitions
Ruby Client Library
Features
Developer host
Calabash server
Simulator
Your app
UIQuery
Run loop:
UIAutomation
Developer host Simulator
Step Definitions
Ruby Client Library
WDA Runner
Features
Calabash server
Your app
UIQuery
FB UIA WDA

42. II. iOS 10 Update:
from UIAutomation WDAto XCTest WDA
Link: https://github.com/facebook/WebDriverAgent
Developer host Simulator
Selenium Driver
Your app
FB UIA WDA
WDA Runner

43. II. iOS 10 Update:
from UIAutomation WDAto XCTest WDA
Link: https://github.com/facebook/WebDriverAgent
Developer host Simulator
Selenium Driver
Your app
FB UIA WDA
Developer host Simulator
Your app
FB XCTest WDA
WDA Runner
Selenium Driver
FB simctl

44. II. iOS Hybrid 2.0
iOS 10 update
Developer host Simulator
Step Definitions
Ruby Client Library
WDA Runner
Features
Calabash server
Your app
UIQuery
FB UIA WDA

45. II. iOS Hybrid 2.0
iOS 10 update
Developer host Simulator
Step Definitions
Ruby Client Library
WDA Runner
Features
Calabash server
Your app
UIQuery
FB UIA WDA
Developer host Simulator
Step Definitions
Ruby Client Library
FB simctl
Features
Calabash server
Your app
UIQuery
FB XCTest WDA

46. III. Summary: multi-desktops vs multi-simulators
Test Suite: 500 tests
Nightly Cucumber Report
Happy morning!
80
data
Mac PRO
Test Suite Run Time
0 min
225 min
450 min
675 min
900 min
1 sim 10 sims 16 sims
75 min
120 min
900 min
Multi
Desktops
Multi
Simulators

47. Developer host Simulator
Step Definitions
Ruby Client Library
FB simctl
Features
Calabash server
Your app
UIQuery
FB XCTest WDA
"Face"book + Cala"bash" = Facebash
III. Summary: current approach

48. - easy to set up Parallel Run on 1 Desktop
- keep using of backdoors
- full access to internal AUT state
- cross iOS app automation
- iOS simulator control with FBSimCtl
- FB WDA supports Selenium protocol
- FB WDA independent of Calabash/Appium
Facebook
Calabash
Calabash
Facebook
Facebook
Facebook
Facebook
III. Summary: PROs

49. - support framework parts: calabash server, FB WDA, fbsimctl, ...
- "poor" documentation
- using of mixed locator strategies
III. Summary: PROs and CONs

51. - Parallel Run in Headless Mode
- Simulator Start/Shutdown/Reset optimization
- Cold/Hot start iOS app with relogin
- record video per the each test in Parallel Run
Optimization

52. Thank you!
v.karanevich@corp.badoo.com