How to collect early cues for winning designs Join UserTesting’s Director of Product, Tanya Koshy, and guest speaker Sr. Analyst at Forrester, Christopher Condo, in a webinar tailored towards product professionals looking to build software faster, safer and in a more controlled manner by removing the guesswork from the process. Attend this webinar to hear the latest findings from Forrester’s recent research on experimentation platforms and learn new ways customer feedback and validation can be leveraged at every stage of development (conception, design, build, launch, production and iteration) to drive faster innovation. Gone are the days of press and play deployments. It’s time product managers learn how to harness real human insights to quickly validate product opportunities and ideas throughout the development cycle. Tanya will dive into direct user testing experimentation platforms and how the latest features can help product managers get to quick answers to support decisions that can influence product direction and improvements and ultimately empower more customers to be part of the value equation. You’ll learn: The 4 key levers of modern application delivery and how to empower your team. How an environment of experimentation can lead to a customer-obsessed organization. How best to roll out products in an agile, automated and controlled manner. Creative, simple methods for testing product ideas and concepts early and often.

1. A Product Manager’s Guide
to Building the Right
Software in Less Time
UserTesting Webinar Featuring Forrester Research
February 27, 2019

2. 2
.
USERTESTING
Tanya Koshy
VP Product Management
UserTesting
Christopher Condo
Senior Analyst
Forrester Research

3. © 2019 FORRESTER. REPRODUCTION PROHIBITED.

4. 4© 2019 FORRESTER. REPRODUCTION PROHIBITED.
 Companies must transform to keep pace with digitally savvy,
restless customers.
 They should reposition their internal structure to better
capitalize on emerging technologies.
 Organizations must respond to a constant barrage of internal
and external disruptions in their respective industries.
Forrester’s predictions for 2019

5. 5© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Transforming the business to be customer focused and
deliver value directly to the customer via digital capabilities.
Digital Transformation:

6. 6© 2019 FORRESTER. REPRODUCTION PROHIBITED.
OLD WAY: Digital Experiences Are Optimized By What’s
Easiest to Execute
?
…The Customer Experience Is An
Afterthought
Did I fill it out
correctly?
Am I getting a
refund?

7. 7© 2019 FORRESTER. REPRODUCTION PROHIBITED.
NEW WAY: Experience is Optimized for the Customer
Back-end capability is exposed via modern architectures
and UIs to Enable Customer-Obsessed CX

8. 8© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Customer Experience: The complete end-to-end user
experience the customer has with your product.

9. 9© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Improving The
Customer
Experience Is
Business Critical

10. 10© 2019 FORRESTER. REPRODUCTION PROHIBITED.
38%
Of developers
use indirect
measurements
(like defects) to
gauge success
Oops!
Something
went wrong.
Source: Forrester Analytics’ Business Technographics Global Developer Survey, 2018

11. 11© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Shutterstock.com 99566564
Maybe We
Need More
Automation?
Maybe We Need
More
Experimentation?

12. 12© 2017 Forrester Research, Inc. Reproduction Prohibited
Customer led
Insights driven
Fast
Connected
Customer-obsessed DNA
FROM
Customer aware
Data rich
Perfect
Siloed
TO

13. 13© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Organizations
Need Product
Managers To
Blend Customer
Insights With
Business
Objectives To
Drive AD&D
Outcomes.

14. 14© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Dev Focuses on
Velocity and
Quality
PM Focuses on
Customer Experience
and Outcomes

15. 15© 2019 FORRESTER. REPRODUCTION PROHIBITED.
A Product-
Centric Team
Has Both Core
Members And
Supporting
Members

16. 16© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Use Experimentation To Capture Customer Insights
And Drive Product Development Direction
Direct User
Testing
A/B Testing Feature
Flags

17. 17© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Direct User Testing
Direct User Testing

18. 18© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Characteristics Of Direct User Testing
› Direct user testing provided the input from real people encouraged
to speak their thoughts while using your application to help you
understand “Why” they prefer one version over another or uncover
issues.
› Components:
• Real people invited to participate in testing
• A lab or platform that allows users to be recorded as they test your app
• A well-thought-out experiment or hypothesis for users to test
• Note: A finished app is not required for subjects to provide meaningful results
Direct User Testing

19. 19© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Direct User Testing (DUT)
Source: “AD&D Pros: Stop Guessing And Start Experimenting”
Direct User Testing

20. 20© 2019 FORRESTER. REPRODUCTION PROHIBITED.
A/B Multivariate Testing
A/B Multivariate Testing

21. 21© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Characteristics Of A/B Multivariate Testing
› A/B Multivariate Testing enables in-production testing of different
versions of an application (work flow, algorithms, search engines,
etc.) to determine a winning strategy within hours or days.
› Components:
• A/B testing logic embedded in code
• A system that controls which populations see Experience A vs. Experience B
• A statistical engine to determine the “winning experience”
A/B Multivariate Testing

22. 22© 2019 FORRESTER. REPRODUCTION PROHIBITED.
A/B Multivariate Testing
Source: “AD&D Pros: Stop Guessing And Start Experimenting”
A/B Multivariate Testing

23. 23© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Feature Flagging
Feature Flagging

24. 24© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Characteristics Of Feature Flagging
› Feature Flagging allows application code to be released in “off”
mode and managed with a dashboard enabling single-trunk
development that enables stabile releases.
› Components:
• If-Then-Else logic embedded in app code
• App or service that communicates value of the feature flag
• Administrative console for managing flags
• Ability to configure which user populations see each feature when switched
“on”
Feature Flagging

25. 25© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Feature Flagging
Source: “AD&D Pros: Stop Guessing And Start Experimenting”
Users Feature Flag Code
Feature Flagging

26. 26© 2019 FORRESTER. REPRODUCTION PROHIBITED.
“This allowed us to deploy
code and feel confident
that customers are seeing
the right features.”
So What Did Developers See After Experimenting?
“We get senior executives
engaged on the release
process because they can
see the data from the
experiments.”
“Now our product manager
can coordinate with
marketing, sales etc. until
they are ready to release.”
“Using feature flags does
helps the team release
faster by helping us push
continuous integration.”
“These tools force us to
align around the customer
experience.”
“We reduced the number
of dependencies, people
involved, and the stress of
making changes. It’s much
easier to iterate and
improve.”

27. 27© 2019 FORRESTER. REPRODUCTION PROHIBITED.
Experimentation Results In Fast Time To Value
› Create a premise to narrow down choices
› Get the pros involved for high-impact changes
› Know that building your own experimentation platform can quickly
become impractical
› Don’t use experimentation as a substitute for tried and true testing
› Understand that experimentation creates technical debt

28. A Product Manager’s Guide
to Building the Right
Software in Less Time
Tanya Koshy
VP Product Management, UserTesting

29. 29
Validate & experiment at every stage

30. 30
Customer
Who is my
customer?
Problem
What is their
problem?
Solution
What’s the
best, lightest
solution I can
build for
them?
Key Product Questions

31. I have an
idea for a
problem
area
I want to
learn more
about my
customers’
needs/pains
in this area
Do people
actually
have the
problem I
think they
have?
Are people
currently
unsatisfied
with their
experience
in this area?
How do
people solve
their
problem
today?

32. 32
How often are you doing X?
Describe your most recent experience doing
X. What worked & what didn’t?
What made doing X easy?
Needs Discovery
Interview
Questions
What are their pain points doing
X? Great way to inspire new ideas
for products / features to build.
What made doing X frustrating/hard?
What did you use to do X?
How similar was your most recent experience
to how you typically do X?
How satisfied are you with X, in general? 1 =
not at all, 5 = very

33. I have a better understanding of their overall experience and
their biggest needs/pain points.

34. Do people
understand the
purpose of the
solution? Do
they find it
valuable?
Does the
solution
actually solve
the problem?
Does it solve it
well?

35. 35
Feature 1
1 - 2 sentence description
Feature 2
1 - 2 sentence description
Feature 3
1 - 2 sentence description
Template for Testing Product Ideas as a Landing Page
Your 1-Line Value Prop Here

36. 36
Spend 1 - 2 mins reviewing the page, share
your thoughts out loud as you do.
What are your initial impressions?
In your own words, based on this page,
describe what this product does?
Great Interview
Questions
Going through the page section by section,
which one is most compelling to you? Why?
Which is least compelling to you? Why?
What must this product do, in addition to
anything you saw here, that would get you to
use it?
How likely would you be to use this product?

37. We have a fine-tuned idea for a solution that customers find valuable.

38. Is the app easy
to use? What is
the first-time
user
experience
like? Is it
compelling?
What do we
need to
improve?

39. We have validated that our solution is usable and have
identified new areas for further improvement.

40. Bring customers into every
stage of development
From needs discovery, to concept
& prototype validation, thru live
site usability. Test mobile apps,
websites, & real world
experiences.
Results Within Hours
Rich video results from your target
customers, answering the
questions you have. Make
confident decisions & share data to
get buy-in across your org.
Same-day Video Interviews
Live Conversation: Schedule
and recruit live interviews in
less than a day. Talk face to
face & share screens.
Your Exact Customer
Easily recruit your target audience
from UserTesting’s diverse, global
panel.
UserTesting

41. 41
Questions?
USERTESTING
If you have any further questions please contact us at: webinars@usertesting.com

42. 42
Thank You!
USERTESTING
If you have any further questions please contact us at: webinars@usertesting.com