How do successful brands drive revenue, attract and retain loyal customers, and get ahead of the competition?

1. WHY APPS
SUCCEEDFour Ways Enterprises Can Overcome Mobile and
Web Testing Challenges to Achieve Digital Excellence
Based on insights from 1,000 enterprise web and mobile professionals

2. 2
“While users demand
new mobile features and
functionality, development and
testing teams are struggling to
keep up.”
How do successful brands drive revenue, attract and retain loyal
customers, and get ahead of the competition?
Delivering exceptional digital experiences across web and mobile channels has become a
critical path to achieving these goals, and separates the leaders from laggards. But creating
a reliable and engaging digital experience for customers is also an increasing challenge for
enterprises. While users demand new mobile features and functionality, development and
testing teams are struggling to keep up with the growing proliferation of mobile devices, the
speed at which new devices and operating system versions are launched, and the variety of
user conditions that need to be tested.
To provide guidance and insight, we sent a survey about digital quality strategies to
enterprise-level web and mobile professionals from various industries and geographies
and received over 1,000 responses. The majority of respondents were in quality assurance,
management and developer roles. Based on the survey results, we’ve outlined four key
obstacles to digital excellence and our recommendations for overcoming them.
Executive Summary
Key takeaways from the survey findings:
• A unified digital development and
testing strategy is imperative,
driving a strong trend toward
responsive websites and a single
unified team for mobile and desktop
web testing.
• Organizations still face significant
challenges releasing new features
and functionality in a timely
fashion and delivering a consistent
experience across devices and
operating systems.
• Greater adoption of test
automation, better testing of real
user conditions, and having an
efficient test lab strategy are major
opportunities for organizations to
improve the digital experience.

3. 3
Digital Users Straddle Multiple Devices
Today’s digital consumer uses multiple devices to pay bills, shop, book flights, watch movies, and more. Forty
percent of eCommerce transactions involve multiple devices along the path to purchase, according to research
and advertising firm Criteo. While the desktop browser is still widely used for common online activities, the rise
of mobile usage in terms of the time spent as part of the digital experience is undeniable.
Desktop is also the dominant choice for many important categories, including business/finance and
entertainment/news.
0 20 40 60 80 100
Source: comScore Media Matrix Multi-Platform, U.S., Dec 2015
69%
58%
51%
46%
44%
39%
38%
36%
23%
21%
12%
8%
6%
62%
64%
77%
79%
88%
92%
94%
56%
61%
54%
49%
42%
31%Portals
Desktop
Business/Finance
Entertainment - News
Health - Information
News/Information
Sports
Retail
Lifestyles
Personals
Social Networking
Online Gaming
Photos
Maps
Share of Content Category Time Spent by Platform
Mobile
0
300,000
600,000
900,000
1,200,000
1,500,000
Growth in Digital Media Time Spent
Source: comScore's "2015 Mobile App Report", Dec. 2015
Jun-2013
476,553 480,967
550,522
118,299
409,847
621,410
778,954
97,440
77,081
Jun-2014 Jun-2015
Desktop
Mobile App
Mobile Browser
TotalMinutes
Increase in time spent since 2013
Desktop
(53.4%)
Mobile App
(90%)
Mobile Browser
(15.5%)

4. 4
The Digital User Experience Is Critical
But no matter what screen a consumer is using, the ability to deliver a reliable, consistent and
engaging user experience is becoming progressively important to any organization with a
digital presence. Overall, 81% of the respondents view the digital experience as crucial
to their business success—58% see it as a critical competitive issue, while 23% view it as a
matter of “do or die.”
Critical to our competitive positioning
Have to do it if we want to survive
Important but not critical
It is a nice-to-have
Not important
�����������58%
3% 1%
23%
15%
“Eighty-one percent of
the respondents view
the digital experience as
crucial to their business
success—58% see it as a
critical competitive issue,
while 23% view it as a
matter of ‘do or die.’”

5. 5
A Unified Strategy
The multi-device behavior of today’s digital user is reflected in the development and testing
strategies of our survey respondents. It’s clear that organizations are preparing themselves
to satisfy users no matter how or where they access digital information. Over three quarters
(76%) of the respondents have both a mobile and web presence, while 15% are mobile-only
and 9% have a web-only presence.
Company size also impacts decisions about multi-device experiences. On average, small
organizations (1-24 employees) were twice as likely than enterprises (more than 1000
employees) to support only a single device experience.
Concurrently, organizations are adjusting how their test teams are structured. The majority
of the survey respondents (59%) said they have a single unified team for both mobile and
desktop web testing. Having one team is traditionally a smaller business approach, but larger
companies are adopting a “one mobile and web team” approach due to the pressure to deliver
digital products faster. This is an encouraging trend given that when mobile and web teams are
united and using the same tools, it’s easier to spot differences across devices and platforms.
Mobile and Web Mobile only (native/hybrid
mobile app, m.dot)
Web only (dot.com)
�����
76%
15%
9%
One team for both mobile
and desktop web testing
59%
Separate mobile and
desktop web teams
41%

6. 6
Digital Excellence:
4 Obstacles and Recommendations
While delivering a reliable and consistent experience is critical to most of the organizations
in our survey, it is also their biggest challenge. Two-thirds of the respondents (67%) view the
delivery of a reliable and consistent digital experience as a major challenge, and 30% see it as
a minor challenge.
There are several obstacles that stand in the way of delivering a superb digital experience,
including the desire to release new functionality faster, the challenge of doing testing that
accurately represents real user conditions on an array of devices and operating systems, and
the need to keep up with the pace of new technologies.
�������67%
30%
3%
Major challenge
Minor challenge
No challenge

7. 7
Obstacle #1:
Releasing new digital functionality faster
Almost every organization responding to our survey is struggling to keep up with users’ high
expectations. As many as 95% of the organizations cited the timely releases of new digital
functionality as either a major (65%) or minor (31%) challenge. This challenge is reflected in
the slow release pace we’re seeing today: the median number of releases in 2015 was just five,
and close to a quarter (24%) of the organizations had only two or fewer releases.
At the same time, organizations realize they need to accelerate the pace of mobile app and
website releases to meet customer expectations. The median number of releases is expected to
jump from five to eight in 2016.
Stepping up the pace of product releases requires organizations to adopt new strategies.
When asked how they plan to increase their product release frequency, the key strategy cited
by respondents was more automation and unit testing, followed by more parallel testing on
multiple devices. (See graph on the next page)
Product releases 2015 Product releases 2016
Source: comScore Media Metrix Multi-Platform & Mobile Metrix, U.S., Total Audience
476,553
bile Browser
0
5
10
15
20
25
30
35
0
5
10
15
20
25
30
35
2 or fewer
24%
15%
31%
24%
16%
15%
35%
20%
12%
9%
3-6 7-12 13-25 26 or more 2 or fewer 3-6 7-12 13-25 26 or more
Median
5
Median
8

8. 8
While organizations see test automation as the primary strategy for faster product releases,
most are not there yet. A majority of organizations (55%) have automated 40% or less of their
mobile testing, and only 23% have automated over 61% of their mobile testing.
The picture is slightly brighter in web testing, where 37% of the organizations have automated
61% or more of their testing.
rowser
0 10 20 30 40 50 60 70 80
More automation and unit testing
Testing on more devices in parallel
Other
Using customer data to do
more targeted testing
More proactive app/web
monitoring after release
66%
56%
40%
35%
8%
Strategies for Improving Frequency of Product Releases
Mobile testing Web testing
476,553
rowser
0
5
10
15
20
25
30
0
5
10
15
20
25
0-20%
30%
20%
19%
24% 24%
13%
25%
22%
16%
7%
21-40% 41-60% 61-80% 81-100% 0-20% 21-40% 41-60% 61-80% 81-100%
What Percentage of Your Mobile/Web Testing Is Done Using Automation?
PercentageofRespondents
Percentage of Automation Percentage of Automation

9. 9
Recommendations
Automation is simply a MUST for your company to keep up with the mobile market. A big bank
or airline can lose hundreds of thousands of dollars if their app is slow to market. Automation
tests can be done quickly on many devices simultaneously through one script, and can be
repeated at no additional cost.
It’s important that test automation be spread across the entire SDLC (software development
lifecycle). For developers, that means automating unit tests and BATs (build acceptance
tests); for QA, that means automating functional tests; and for IT ops, that means assuring
performance and integration testing.
A good question to ask at the start is: What should I be automating? Organizations often do
not choose the right tests to automate, resulting in lost development time and a reliance
on manual testing. The key to mobile test automation is to start with a small number of test
cases, automate them, and assure they can be executed in parallel and unattended. Only then
should you invest more in your test automation suite. We recommend that 70% of your testing
be automated.
The ultimate benefits of strong automation are that your team gets more tests executed
earlier, meets release deadlines, and prevents defects from making it to customers’ screens.
To learn more, listen
to the webinar, 7 Ways
to Overcome Test
Automation Obstacles
Sixty-six percent of
respondents believe
that doing more
automation is the
key to faster product
releases.

10. 10
Obstacle #2:
Accurately testing for real user environments
It was encouraging to see that a majority of respondents (79%) are mimicking real user
environments in their mobile testing.
The top two mobile user conditions that respondents are testing against are switching network
conditions and apps running in the background. Other commonly tested conditions include
call/text message interruptions and low memory.
Yet at the same time, the vast majority (96%) of the organizations are still challenged to
guarantee that testing accurately represents real user environments – with 69% viewing it as a
major challenge and another 27% calling it a minor challenge.
bile Browser
0 10 20 30 40 50 60 70 80
Apps running in the background
Call and text message interruptions
Low memory
Spikes in mobile traffic
Dying battery
Switch networks - from
3G to 4G to Wi-Fi etc.
Other
71%
71%
56%
51%
43%
40%
8%
What User Conditions Do You Test Against?

11. 11
Recommendations
Step one in testing against real world user conditions is to identify your target users (aka
“personas”) through customer data analytics. Getting consistent customer usage data relies
on regular communication between technical and marketing teams.
The next step is integrating these user traits into testing.
Requirements for a lab that tests for user conditions include:
• Ability to insert tests for various network conditions, apps running in the background,
phone call interruptions, and more
• Support for different geographies to test for relevant users in relevant countries
• Support for any device and operating system as they become available
Once the lab is ready, DevTesters can add the relevant user conditions to existing functional
tests in various ways, including manual testing, data-driven testing or through automated
test code. We recommend automation.
After defining target personas and implementing test code, you’ll need an actionable test
report that provides valuable insights into the real user experience of your customers. The
report should include: network logs, device and app memory consumption details, and timers
showing the duration of certain user actions.
For more details on how to test for real users,
download the white paper: 3 Keys for Mobile
App Testing for Real User Conditions.

12. 12
Obstacle #3:
Staying up on the latest devices and operating systems
Organizations are finding it difficult to keep up with mobile market changes and the pace of technology. Ensuring that testing
accurately delivers a reliable experience on new devices and operating systems is a major challenge for 64% of survey respondents.
Responsive websites play a pivotal role in addressing this challenge – 88% of the organizations either already have a responsive
website or are in the process of developing one.
The trend toward responsive websites is driven by: the quest to provide a more consistent digital experience across devices (46% of
respondents) as well as the need to guarantee mobile websites work on new devices as they become available (35% of respondents).
Visual testing and navigation testing are the most common strategies for responsive website testing (see second graph below).
obile Browser
0
10
20
30
40
50
60
70
80
Visual testing Navigational
testing
Client-side
performance
Testing
User condition
testing
76%
72%
61%
57%
Mobile Browser
0
10
20
30
40
50
60
Yes, our websites
are responsive
We plan to in the
next 6 months
We plan to in the
next 12 months
We have no
plan to do
responsive design
We are currently
developing a
responsive
website
59%
14%
8% 7%
12%
Fifty-nine percent
of organizations
currently have
responsive websites
and 29% are adopting
responsive web over
the next 12 months.

13. 13
Recommendations
A key way to prepare for new devices and operating systems is to create a market calendar
—or use the mobile and web browser market calendars from our latest Digital Test Coverage
Index report—and plan accordingly.
With all the new releases in a given year, you should keep a close eye on the market. Include
beta testing of upcoming OS and browser versions and plan on a device lab refresh to
accommodate new devices.
As we mentioned, responsive web design (RWD) is one great way to make sure your digital
products will work on new devices. RWD done right will deliver a seamless digital experience
on every screen, but must include rigorous testing.
Three RWD testing scenarios to keep in mind:
• Visual testing to make sure content looks right on every screen
• Client-side performance testing to measure the time it takes website objects to
render on different screens
• Navigation testing to verify that the website performs expected functions correctly on
all devices
Not all companies have multiple approaches to delivering content to a variety of devices
—particularly with responsive websites—yet the need for a consistent user experience
and supporting analytics is key to success. Dig in to customer usage data for clues on how
to prioritize your testing strategy. For instance, if 80% of your mobile traffic is from iOS
devices, start testing on the upcoming iOS 10 beta version the minute it’s available.
OS
PROJECTED
TYPICAL
ADOPTION OF
MAJOR OS
DEVICES
JANUARY FEBRUARY MARCH APRIL MAY JUNE JULY AUGUST SEPTEMBER OCTOBER NOVEMBER DECEMBER
Honor Holly 2
Plus
Media Pad M2 P9
Nexus 7P
HTC One M10
LG K7
LG G5 LG V11
Nexus 6X
OnePlus 3
Surface Phone
Droid Turbo 3Xiaomi Redmi 3 Xiaomi Mi 5
iOS 10
release
iOS 10.0.1
release
Android 6.1
Android 6
2% share
Android 6
10% share
Android 6
15% share
iOS 10.2
release
Windows 10
iPad Air 3
iPad Pro 9.7"
iPhone SE
iOS 10
announcement
Android 7
release
iOS 10.1
release
iOS 9.2.1 iOS 9.3 Beta
public availability
iOS 9.3.2iOS 9.3.2 BetaiOS 9.3
iOS 9.3.1
iOS 9 = 77%
iOS 8 = 17%
Other = 6%
iOS 10 = 35%
iOS 9 = 60%
Other = 5%
iPhone 7
iPhone 7 Plus
iPad Mini 5
Watch 2
iPad Pro 2
Live G500TG
Galaxy S7
Galaxy S7 Edge
Galaxy Note 4
Upgrade available
Android 6.0
Galaxy S4/S5
Upgrade available
Android 6.0
Galaxy Note
S6/Edge/Note 5
Upgrade available
Android 6.0
Sony Xperia X
Galaxy A9 Note 6
Sony Xperia X
HP Elite X3
(Win 10)
HTC One X9
Mobile Market Calendar 2016
Be sure to plan ahead to get maximum test coverage.
Our Ultimate Digital
Test Coverage
Toolkit offers three
free tools to help
you create a test
coverage strategy.

14. 14
Obstacle #4:
Building a more efficient lab for mobile testing
Companies have options when it comes to mobile testing—from using homegrown mobile test labs to using
cloud-based or on-premise commercial solutions to offload device ownership and improve reliability.
While commercial mobile testing solutions have become more robust and offer significant capabilities,
many organizations – including 45% of our survey respondents – continue to run in-house mobile test labs
despite challenges.
For companies that use their own lab, keeping up with frequent browser and device releases is the top
challenge. Maintaining the test lab and keeping it up-to-date and secure is another significant issue, with
some of the survey respondents reporting they actually devote more resources to lab maintenance than to
their test codebase.
0 10 20 30 40 50
New browsers and devices are released
to market faster than we can keep up
We don’t have a dedicated team to manage resources
and update our lab quickly enough
We don’t have the expertise in-house to keep our
internal lab up to date
We focus more on test lab
maintenance than our test codebase
Other
44%
29%
11%
8%
8%
The Top Challenges Teams Face When Running In-House Testing Labs

15. 15
Recommendations
It may seem convenient to buy and test manually on mobile devices in-house, but in reality
it’s difficult and expensive. You will have to purchase and update the device, replace it if it’s
lost, fix it if the screen cracks, ask IT to fix driver conflicts and allocate desktops to connect
to the devices. The annual total-cost-of-ownership to manage a single mobile device in-
house is $6,500!
We recommend testing on real devices, but for the reasons stated above we do not
recommend buying them yourself and testing them in-house.
To test your apps or websites as your users would experience them on a mix of real devices,
networks and locations, a cloud-based lab-as-a-service that’s available 24/7 and can scale
as needed is the optimal choice.
A cloud-based lab gives teams in different geographies remote access to the same devices,
makes new devices and OSes available immediately and allows unattended automation
testing because devices are always accessible in a secure environment.
“It may seem convenient to buy and test
manually on mobile devices in-house, but in
reality it’s difficult and expensive.”
Read the new
Forrest Wave report
to see who’s leading
the way in mobile
front-end test
automation tools.

16. 16
Conclusion
A seamless, cross-device digital experience is increasingly vital to companies that want to
engage with customers effectively. This requires organizations of all sizes to quicken the pace
of development and release without decreasing the quality of those experiences.
Most organizations still have a long way to go toward automating the testing of mobile
and web applications, accurately representing the user environments, and adopting a test
lab strategy that can scale to meet today’s needs. Yet by overcoming these challenges,
an unprecedented opportunity will open up for companies to deliver memorable digital
experiences to their customers.

17. 17
Survey Methodology and Demographics
For this report, Perfecto conducted a survey in April 2016 and received over 1,000 responses from enterprise-level web and mobile
professionals representing a variety of roles, company sizes, industries and geographies.
���������58%
15%
20%
4%
5%
Quality Assurance
Management
Development
DevOps
Other
���������46%
19%
30%
1%
4%
North America
Asia/Pacific
Europe/Middle
Central/South
Africa
High
Tech
FinancialServices
Telecom
m
unications
Healthcare
Advertising&
Marketing
Retail
Media&
Entertainm
et
Manufacturing
PublicSector
Other
21%
19%
10% 10%
9%
8%
5%
4% 4%
11%
0
5
10
15
20
25
Role
The majority of survey respondents (56%) are in quality assurance
roles, while another 20% hold management positions.
Geography
46% of respondents are based in North America, 30% in Asia/
Pacific, 19% in Europe/Middle East and 4% in Central/South
America.
Company Size
Close to half of respondents come from large companies with over
1,000 employees.
Industry
Survey respondents come from a wide range of industries, led by
high tech and financial services.
58%
15%
20%
4%
5%
1-24
14%
12% 11%
13%
49%
25-100 101-250 251-1000 > 1000
0
10
20
30
40
50

18. 18
See which devices should be covered
in your test plan by visiting:
www.perfectomobile.com/testoptimizer
© 2016 Perfecto Mobile Ltd.
www.perfectomobile.com
781.205.4111
About Perfecto
Perfecto enables exceptional digital experiences. We help you transform your business and strengthen every digital interaction with a quality-first approach to
creating web and native apps, through a cloud-based test environment called the Continuous Quality Lab. The CQ Lab is comprised of real devices and real end-
user conditions, giving you the truest test environment available.
More than 1,500 customers, including 50% of the Fortune 500 across the banking, insurance, retail, telecommunications and media industries rely on Perfecto
Mobile to deliver optimal mobile app functionality and end user experiences, ensuring their brand’s reputation, establishing loyal customers, and continually
attracting new users. For more information about Perfecto Mobile, visit perfectomobile.com, join our community, or follow us on Twitter at @PerfectoMobile.