Watch a live presentation at http://offer.bitbar.com/parallel-test-runs-with-appium-on-real-mobile-devices Appium is an open source test automation framework and currently one of the hottest framework for mobile app, game and web testing. In this webinar, we’ll discuss about the best practices, how to use Appium for different types of apps, games and web apps, and how to use all new features around it – the parallel test runs, image recognition and more! Stay tuned and join our upcoming webinars at http://bitbar.com/testing/webinars/

1. 12 May 2015
Ville-Veikko Helppi
Technical Product Manager
ville-veikko.helppi@bitbar.com
Parallel Test Runs with
Appium on Real Mobile
Devices
W E B I N A R

2. Agenda
• Testdroid Update
• Statistics from Q1'2015
• Testing and Mobile Test Automation
• Appium – Client vs. Server Side
Execution
• Demonstration
• Q&A

3. TESTDROID UPDATE

4. Public Device Cloud
on-demand devices
(multitenant)
Mobile app testing
on thousands of real
Android and iOS
devices hosted by
Bitbar
Private Device Cloud
Reserved Devices
Hosted by Bitbar in
the US and/or Europe
Devices chosen by
and reserved only for
the Customer
On-Premise
Device Cloud
Automated mobile
app testing devices
hosted by the
customer, usually
30-500 devices
1 Product – 3 Deployment Options
Testdroid Cloud Testdroid Enterprise Testdroid PrivateCloud

5. New Pricing Plans for Cloud
*) = Prices for annual commitment

6. Appium
• Client-Side Execution
• Server-Side Execution

7. Q1'2015 STATISTICS

8. Q1'2015 – Test Run Stats

9. Why Apps Fail?

10. iOS 8.0
iOS 8.0.2
iOS 8.1
iOS 8.1.1
iOS 8.2
iOS 8 ALL
KitKat 4.4
KitKat 4.4.2
KitKat 4.4.3
KitKat 4.4.4
KitKat ALL
Lollipop 5.0
Lollipop 5.0.1
Lollipop 5.0.2
Lollipop ALL
0 5 10 15 20 25 30 35
Failed test runs. Percentage (%).

11. Some Observations
1
2

12. Some Observations
3
4

13. Some Observations
5
6

14. Android Fragmentation 2014
by Open Signal report July 2014

15. How Many Devices is Enough?
~90%
market coverage can
be achieved with
128
devices
~20%
market coverage can
be achieved with
12
devices
US Market
25 Android devices
= ~2/3 market
Global Market
60 Android devices
= ~1/2 market

16. TESTING AND MOBILE TEST AUTOMATION

17. OS
versions
Chipsets
CPU +
GPU
Tens of
OEMs
Memory
Displays
(resolutions,
physical hw)
OEM
mods
Other
hardware
(connectivity
calibration)
Relation
to other
software
Where Test Automation Can Help

18. Correct behaviour
across platforms and
browsers
Integration with web
back-ends
Typically need
to fully utilize
HW
(CPU+GPU)
Resource (e.g.
battery)
consumption
OpenGL ES 2/3
Functionality
and usability
Screen
orientations,
connectivity,
user profiles
Robustness
Robustness and
security!
Brand
Compliances,
verification with back-
ends and data
Different Mobile 'App Verticals'

19. Manual Testing is not Agile

20. Test Automation is Agile

21. Why Real Devices Are Must-to-Have
• Emulators cannot help you to test...
• User Experience
• Usability
• Hardware
• Software
• Infrastructure
0 % = the percentage of your app users
that use emulator to run your app!

22. Manual vs. Automation
Manual Automation

23. Manual vs. Automation
Smaller coverage, More
money burnt & time
wasted, Error-prone
Manual Automation
Large
coverage,
quickly
completed,
Less money &
time wasted,
Exact results.

24. APPIUM
– CLIENT VS. SERVER SIDE EXECUTION

25. Family Tree of Android Test
Automation Frameworks
JUnit
Android Instrumentation Framework
Robotium
Espresso and
Espresso v2
uiautomator
Appium
ExtSolo
Calabash

26. What Framework Works You The Best?
Robotium uiautomator Espresso Appium Calabash
Android Yes Yes Yes Yes Yes
iOS No No No Yes Yes
Mobile web Yes
(Android)
Limited to x.y
clicks
No Yes
(Android &
iOS)
Yes
(Android)
Scripting
Language
Java Java Java Almost any Ruby
Test creation
tools
Testdroid
Recorder
UI Automator
viewer
Hierarchy
Viewer
Appium.app CLI
Supported
API levels
All 16 => 8, 10, 15- All All
Community Contributors Google Google Active Pretty quiet

27. Selenium & Appium

28. Client Side Appium at
Testdroid CloudTest Script
Test Case
Desired
Capabilities
{
“device”: “Android”,
“app”: “/Users/user/ApiDemos.apk”
“app-package”: “com.example.android.apis”
“app-activity”: “.ApiDemos”
}

29. Test Script
Test Case
Desired
Capabilities
WebDriver
http://localhost:4723/wd/hub
Appium Server
4723
Device
Localhost
(DesiredCaps)

30. http://localhost:4723/wd/hub
*Testdroid Caps
http://appium.testdroid.com/wd/hub
(DesiredCaps)
Test Script
Test Case
Desired
Capabilities
WebDriver
From Localhost to Testdroid

31. Client Side Execution
Go to
cloud.testdroid.com

32. Client Side Execution
Get a
Device Name
Go to
cloud.testdroid.com

33. Client Side Execution
Add Testdroid Desired
Caps to test script
{
“testdroid_username”: “user@domain.com”,
“testdroid_password”: “p4s$w0rd”,
“testdroid_project”: “My First Project”,
“testdroid_testrun”: “Test 1”,
“testdroid_device”: “iPad Mini 7.0.4 A1432”,
“testdroid_app”: “http://domain.com/app_v1.ipa”
.
.
“app”: “com.bitbar.testdroid.BitbarIOSSample”
}
Get a
Device Name
Go to
cloud.testdroid.com

34. Client Side Execution
driver = webdriver.Remote("http://appium.testdroid.com/wd/hub", desired_caps);
Point the Webdriver to
http://appium.testdroid.com/wd/hub
Add Testdroid Desired
Caps to test script
Get a
Device Name
Go to
cloud.testdroid.com

35. Client Side Execution
Run the Test ScriptGet Results from
Testdroid Cloud
Point the Webdriver to
http://appium.testdroid.com/wd/hub
Add Testdroid Desired
Caps to test script
Get a
Device Name
Go to
cloud.testdroid.com

36. Client Side Execution
Pull the Results from
the Result URL
driver = webdriver.Remote("http://appium.testdroid.com/wd/hub", desired_caps);
Run the Test ScriptGet Results from
Testdroid Cloud
Point the Webdriver to
http://appium.testdroid.com/wd/hub
Add Testdroid Desired
Caps to test script
Get a
Device Name
Go to
cloud.testdroid.com

37. Multiple Devices – Client Side
python testscript.py –device <devicename>1
python testscript.py –device <devicename>2
python testscript.py –device <devicename>n
Test Script
Test Cases
Instigator Script
deviceArray=
[
“iPad 4 6.0.1 A1458”,
“iPad mini 7.0.4 A1432”,
.
.
.
“iPhone 4S 6.1.3 A1387”,
]

38. 21
3
4
5
6
7
WebDriver Session Request
@http://appium.testdroid.com/wd/hub
WebDriver Session Response
Testrun
Configure Project
Appium Ready
Wait for Device to
Become Available
Device 1 Device 2 Device 3
Session
Map
Proxy
Device Cluster
Start Appium
Desired Capabilities, .apk / .ipa
SessionID
SessionID
Sessionid
Test
Script
Appium
Broker
Client Side – Behind the Scenes

39. Introducing "Server Side Appium"

41. Demonstration
Appium with Real Devices using Image
Recognition for Hill Climb Racing

42. Setup
• Using real Android devices at Testdroid Cloud
• Parallel test runs without a need to configure desired
capabilities
• Device groups (= set of devices used for runs) can be
manually created and configured

43. File Structure
• pom.xml (maven)
• testdroid.properties (overwritten after submitted to Cloud)
• run-test.sh (shell script for execution)
• image files

48. 15 seconds

50. Image Recognition
• Resolution agnostic implementation
• Identifies stretched and rotated images as well

51. Let's See How it Works!

53. THANK YOU!
More information about frameworks, devices
and testing at www.testdroid.com