Am I testing on the right platforms? This is the question that app developers and testers ask themselves every day. As more brands connect with users digitally, it’s getting harder to ensure a great user experience across mobile, web & wearables. In this webinar, you’ll learn how to create an effective and customized test coverage strategy. We’ll describe how our new Test Coverage Toolkit’s combination of resources, recommendations and research can help you: Know exactly what mix of mobile devices and browsers to test on Keep up with the latest mobile devices, operating systems and browsers Learn how to setup and maintain a robust and scalable test lab Deliver a great digital user experience across devices

1. 5/26/2016 1© 2016, Perfecto Mobile Ltd. All Rights Reserved.
How to Create the Right Mobile &
Web Test Strategy
May 2016
Eran Kinsbruner
Director, Tech. Evangelist
Perfecto
Patrick McCartney
Senior Sales Engineer
Perfecto

2. Some housekeeping before we start
5/26/2016 2© 2016, Perfecto Mobile Ltd. All Rights Reserved.
Submit questions
using the Q&A panel.
Participate in the
polls.
Watch your email for
a copy of the
recording and slides.
Complete the survey
following the
presentation.

3. Agenda for Today
5/26/2016 3© 2015, Perfecto Mobile Ltd. All Rights Reserved.
Web
Coverage
Appendix
Best
Practices
The
Optimizer
Index
Intro
Mobile
Coverage
Method
The
Challenges

4. The Test Coverage Conversation – What To Answer?
Which Devices & Browsers?
Which OS
Versions?
How Many
platforms are
enough?
When should I
refresh my test
lab?
Who is in
charge of
coverage?
How to size my
lab across
projects and
teams?
5/26/2016 4© 2016, Perfecto Mobile Ltd. All Rights Reserved.

5. Introducing The – Digital Test Coverage Toolkit
5/26/2016 5© 2015, Perfecto Mobile Ltd. All Rights Reserved.

6. Test Coverage Optimization Process
2. Lab Sizing
Teams Projects
SDLC
Cycle
Requirements
How many?
v
Me
Analytics
Market
My Space
Popular & emerging
1. Coverage Mix
Industry trends & insights
Which?

7. Test Coverage Optimization Process
Me
Analytics
Market
My Space
Popular & emerging
Industry
Current state
Internal
External
Future state
The
Index
The
Optimizer

8. What does it take to cover the
REAL end user experience?
v
Device & Platform
Model
OS Version
Screen Size
browser
v
Environment
Location
Network
Phone events
Other apps
v
Conditions
Signal
Load
CPU

9. What’s new?
• Brazil, China, Netherlands
• Usage vs. Purchase Model
Digital Test Coverage Index 4th Edition

10. 2016 Calendar and Forecast
5/26/2016 10© 2015, Perfecto Mobile Ltd. All Rights Reserved.

11. Market Events Which Impact Your Test Coverage Decisions
11© 2015, Perfecto Mobile Ltd. All Rights Reserved.
&

12. Mobile Test Coverage Method

13. Mobile Device Coverage - Attributes
• Device and OS popularity (market share)
• Screen sizes, resolution and other screen attributes such as pixel per inch
(PPI)
• Device age (launch date)
• New and trending devices and platforms
• Operating system version update rate (e.g. reference devices like Android
Nexus get a higher score)
• Unique device properties important for testing purposes – chipset, CPU,
memory
• Audience demographics
5/26/2016 13© 2015, Perfecto Mobile Ltd. All Rights Reserved.
Reference
Devices/OS
Popular
Devices/OS
Legacy
Devices/OS
Emerging
Devices/OS

14. Reading The Index
Recommended
OS version
Pixels per Inch
Trend
Types
52%
35.5%

15. Recommended OS – Usage & Quality Rationale
Usage &
Quality
Issues

16. The Optimizer
5/26/2016 16© 2015, Perfecto Mobile Ltd. All Rights Reserved.

17. Optimizer Live DEMO
5/26/2016 17© 2015, Perfecto Mobile Ltd. All Rights Reserved.
LIVE DEMO

18. Test Coverage Recommendations
 Leverage market calendars and stay updated
 Refresh/Review your test lab once a quarter
 Beta versions are no longer an option
 Chrome, FireFox, iOS
 Leverage analytics but validate and calibrate using market trends
 Real user conditions extends your test coverage
 Assign an owner for digital test coverage but keep the conversation open to all
teams
 Mix your mobile test lab with the 4 device types
5/26/2016 18© 2015, Perfecto Mobile Ltd. All Rights Reserved.

19. Digital Test Coverage Guide
5/26/2016 19© 2015, Perfecto Mobile Ltd. All Rights Reserved.

20. Lab Sizing
5/26/2016 20© 2015, Perfecto Mobile Ltd. All Rights Reserved.

21. QUALITY
VELOCITY
16 DEVICES
REQUIRED COVERAGE
2 WEEK
SPRINTS
GOAL: 400
REGRESSION TESTS
TO RUN IN 48 HOURS
ARE 16 DEVICES ENOUGH TO COMPLETE
FULL REGRESSION IN 48 HOURS?
Quality vs. Velocity

22. COVERAGE
CAPACITY
EACH DEVICE EXECUTING 400 TEST CASES
48 HOURS
DESIRED REGRESSION TIME
32 DEVICES REQUIRED FOR FULL COVERAGE IN 48 HOURS
ACTUAL REGRESSION TIME
16 X
Meeting Velocity Goals
10 MIN * 400 TEST CASES = 67 HOURS (2.8 DAYS)
67 HOURS – 48 HOURS = 19 HOURS
19 HOURS / 48 HOURS = 0.4 (1 ADDITIONAL DEVICE PER DEVICE TYPE)
2X DEVICES REQUIRED
2.8 DAYS

23. Balancing Velocity and Coverage
Apple iPhone 6
Apple iPhone 6 Plus
Apple iPad Air 2
Apple iPhone 5S
Apple iPhone 6S
Apple iPad mini
Apple iPad 2
Apple iPhone 6S Plus
Primary
{
{Secondary
Samsung Galaxy S6
Samsung Galaxy S5
Samsung Galaxy S4
Samsung Galaxy Note 5
Google Nexus 5X
HTC One (M9)
LG G4
Samsung Galaxy Tab S2
Primary
{Secondary
{
iOS Android
75 High Priority Tests
200 Medium Priority Tests
125 Low Priority Tests
{ }RUN ON PRIMARY
DEVICES (400 TESTS)
RUN ON
SECONDARY
DEVICES (275 TESTS)

24. PRIMARY
25 TOTAL DEVICES FOR OPTIMIZED COVERAGE
PARALLEL
CAPACITY
10 MIN * 400 TEST CASES = 67 HOURS (2.8 DAYS)
67 HOURS – 48 HOURS = 19 HOURS
19 / 48 = 0.4 (1 ADDITIONAL DEVICE PER DEVICE TYPE)
2X DEVICES REQUIRED FOR HIGH PRIORITY TESTS
10 MIN * 275 TEST CASES = 46 HOURS (1.9 DAYS)
7 SECONDARY DEVICES
SECONDARY
Optimized Coverage
18 X
7 X
←

25. Q & A
5/26/2016 25© 2016, Perfecto Mobile Ltd. All Rights Reserved.

26. Practical Toolkit for Webinar Attendees
5/26/2016 26© 2015, Perfecto Mobile Ltd. All Rights Reserved.
perfectomobile.com/coveragetoolkit

27. Thank You