For more information about UserVoice: www.uservoice.com Join Richard White, Founder and CEO of UserVoice, as he dives into how to make data driven product decisions to help inform what to build in your product.

1. Data-Informed Product Decisions
Using Feedback to De-risk your Roadmap
Richard White, Founder & CEO
1

2. 2 © Copyright 2016 UserVoice. Confidential and proprietary information.
90% of product management is building
the right things in the right order

3. Step 1 - Define Your Goals
Increase Win-Rate? Reduce Churn?

4. Step 2 - Estimate Impact Against Goals
Goal A Goal B Effort Confidence Score Accounts % Account $ %
Feature A 0 2 2 50% 0.5 30 15% $300,000 19%
Feature B 0 2 2 75% 0.8 3 2% $300,000 19%
Feature C 1 1 1 50% 1.0 60 30% $600,000 38%
Feature D 2 1 4 90% 0.7 90 45% $900,000 56%

5. Step 2 - Estimate Engineering Effort
Goal A Goal B Effort Confidence Score Accounts % Account $ %
Feature A 0 2 2 50% 0.5 30 15% $300,000 19%
Feature B 0 2 2 75% 0.8 3 2% $300,000 19%
Feature C 1 1 1 50% 1.0 60 30% $600,000 38%
Feature D 2 1 4 90% 0.7 90 45% $900,000 56%

6. Step 3 - Estimate Confidence
Goal A Goal B Effort Confidence Score Accounts % Account $ %
Feature A 0 2 2 50% 0.5 30 15% $300,000 19%
Feature B 0 2 2 75% 0.8 3 2% $300,000 19%
Feature C 1 1 1 50% 1.0 60 30% $600,000 38%
Feature D 2 1 4 90% 0.7 90 45% $900,000 56%

7. Step 4 - Score & Rank
Goal A Goal B Effort Confidence Score Accounts % Account $ %
Feature C 1 1 1 50% 1.0 60 30% $600,000 38%
Feature B 0 2 2 75% 0.8 3 2% $300,000 19%
Feature D 2 1 4 90% 0.7 90 45% $900,000 56%
Feature A 0 2 2 50% 0.5 30 15% $300,000 19%
(GOAL A + GOAL B) x CONFIDENCE / EFFORT = SCORE

8. (REVENUE LIFT + REFERRAL LIFT) x CONFIDENCE
EFFORT
GOAL A GOAL B

9. 100% BEHAVIORAL 100% QUALITATIVE

10. 100% BEHAVIORAL 100% QUALITATIVE
Most Companies

11. 100% BEHAVIORAL 100% QUALITATIVE
Most Companies
BEHAVIORAL
ANALYTICS
CUSTOMER
FEEDBACK

12. Step 4 - Score & Rank
Goal A Goal B Effort Confidence Score Accounts % Account $ %
Feature C 1 1 1 50% 1.0 60 30% $600,000 38%
Feature B 0 2 2 75% 0.8 3 2% $300,000 19%
Feature D 2 1 4 90% 0.7 90 45% $900,000 56%
Feature A 0 2 2 50% 0.5 30 15% $300,000 19%
(GOAL A + GOAL B) x CONFIDENCE / EFFORT = SCORE

13. Step 5 - Connect to Customer Feedback
Goal A Goal B Effort Confidence Score Customers % Account $ %
Feature C 1 1 1 50% 1.0 60 30% $600,000 38%
Feature B 0 2 2 75% 0.8 3 2% $300,000 19%
Feature D 2 1 4 90% 0.7 90 45% $900,000 56%
Feature A 0 2 2 50% 0.5 30 15% $300,000 19%

14. De-risk #1: Do asks back up estimates?
Goal A Goal B Effort Confidence Score Customers % Account $ %
Feature C 1 1 1 50% 1.0 60 30% $600,000 38%
Feature B 0 2 2 75% 0.8 3 2% $300,000 19%
Feature D 2 1 4 90% 0.7 90 45% $900,000 56%
Feature A 0 2 2 50% 0.5 30 15% $300,000 19%

15. De-risk #1: Do asks back up estimates?
Goal A Goal B Effort Confidence Score Customers % Account $ %
Feature C 1 1 1 50% 1.0 60 30% $600,000 38%
Feature B 0 2 2 75% 0.8 3 2% $300,000 19%
Feature D 2 1 4 90% 0.7 90 45% $900,000 56%
Feature A 0 2 2 50% 0.5 30 15% $300,000 19%
Why are we so confident in the impact with so few customer asks?

16. Step 6 - Add in CRM Data
Goal A Goal B Effort Confidence Score Customers % Customer $ %
Feature C 1 1 1 50% 1.0 60 30% $600,000 38%
Feature B 0 2 2 75% 0.8 3 2% $300,000 19%
Feature D 2 1 4 90% 0.7 90 45% $900,000 56%
Feature A 0 2 2 50% 0.5 30 15% $300,000 19%
Ah, that’s because these are our largest Enterprise customers!
Are we overestimating the impact?

17. Step 7 - Add in CRM & NPS Data
Reduce
Churn Goal B Effort Confidence Score
Detractor &
Churned
Customers %
Detractor &
Churned
Customer $ %
Feature C 1 1 1 50% 1.0 5 10% $100,000 10%
Feature B 0 2 2 75% 0.8 0 0% $0 0%
Feature D 2 1 4 90% 0.7 60 40% $650,000 50%
Feature A 0 2 2 50% 0.5 12 9% $90,000 5%

18. De-risk #2: Do asks back up impacts?
Reduce
Churn Goal B Effort Confidence Score
Detractor &
Churned
Customers %
Detractor &
Churned
Customers $ %
Feature C 1 1 1 50% 1.0 5 10% $100,000 10%
Feature B 0 2 2 75% 0.8 0 0% $0 0%
Feature D 2 1 4 90% 0.7 60 40% $650,000 50%
Feature A 0 2 2 50% 0.5 12 9% $90,000 5%
Are we sure of our estimate for C to Reduce Churn? What about A?

19. De-risk #3: Confirm with Accounts
Goal A Goal B Effort Confidence Score Customers % Customers $ %
Feature C 1 1 1 50% 1.0 60 30% $600,000 38%
Feature B 0 2 2 75% 0.8 3 2% $300,000 19%
Feature D 2 1 4 90% 0.7 90 45% $900,000 56%
Feature A 0 2 2 50% 0.5 30 15% $300,000 19%
Before we build C let’s review the feedback from those 60 accounts,
contact them to give feedback on solution mock ups, beta, etc.

20. Takeaways
Use a goal-based scoring framework for making
product decisions
- Make sure that decision framework is consistent and understood across
teams. AKA Show Your Work
- Use your database of feedback to de-risk functionality you put on at the
top of the TODO list.

21. 21 © Copyright 2016 UserVoice. Confidential and proprietary information.
For more details
read the eBook
How do we build that comprehensive database of customer
feedback that informs good product management?

23. Need to tap into both direct & indirect
feedback channels

24. Need to tap into both direct & indirect
feedback channels

25. Getting Feedback
from Users

26. Feedback directly from end-users
Methods:
NPS, Idea Boards, In-app Solicitation
Key Challenge: Low Response Rates
Feedback from only a small vocal minority

27. Feedback directly from end-users
Methods:
NPS, Idea Boards, In-app Solicitation
Key Challenge: Low Response Rates
Feedback from only a small vocal minority
CAUSES
- Lack of awareness - Poor promotion / visibility

28. Feedback directly from end-users
Methods:
NPS, Idea Boards, In-app Solicitation
Key Challenge: Low Response Rates
Feedback from only a small vocal minority
CAUSES
- Lack of awareness - Poor promotion / visibility
- Too much work to give feedback

29. Feedback directly from end-users
Methods:
NPS, Idea Boards, In-app Solicitation
Key Challenge: Low Response Rates
Feedback from only a small vocal minority
CAUSES
- Lack of awareness - Poor promotion / visibility
- Too much work to give feedback
- No value for giver - “What’s in it for me”

30. Feedback directly from end-users
Methods:
NPS, Idea Boards, In-app Solicitation
Key Challenge: Low Response Rates
Feedback from only a small vocal minority
CAUSES
- Lack of awareness - Poor promotion / visibility
- Too much work to give feedback
- No value for giver - “What’s in it for me”

31. Feedback directly from end-users
Methods:
NPS, Idea Boards, In-app Solicitation
Key Challenge: Low Response Rates
Feedback from only a small vocal minority
CAUSES
- Lack of awareness - Poor promotion / visibility
- Too much work to give feedback
- No value for giver - “What’s in it for me”

32. Feedback directly from end-users
Methods:
NPS, Idea Boards, In-app Solicitation
Key Challenge: Low Response Rates
Feedback from only a small vocal minority
SOLUTION
- Promote that you’re listening - “We want your feedback
to help us improve the product. Click here”

33. Feedback directly from end-users
Methods:
NPS, Idea Boards, In-app Solicitation
Key Challenge: Low Response Rates
Feedback from only a small vocal minority
SOLUTION
- Promote that you’re listening - “We want your feedback
to help us improve the product. Click here”
- Single question survey if possible

34. Feedback directly from end-users
Methods:
NPS, Idea Boards, In-app Solicitation
Key Challenge: Low Response Rates
Feedback from only a small vocal minority
SOLUTION
- Promote that you’re listening - “We want your feedback
to help us improve the product. Click here”
- Single question survey if possible
- Explain what the user gets in return - Ideally influence
and not prizes. Set expectations and close the loop.

35. Feedback directly from end-users
Methods:
NPS, Idea Boards, In-app Solicitation
Key Challenge: Low Response Rates
Feedback from only a small vocal minority
SOLUTION
- Promote that you’re listening - “We want your feedback
to help us improve the product. Click here”
- Single question survey if possible
- Explain what the user gets in return - Ideally influence
and not prizes. Set expectations and close the loop.

36. Getting Feedback
from Internal Teams

37. Feedback from customer teams
Methods: Meetings, Spreadsheets
Sales Feedback Success Feedback
1. This thing
2. That thing
3. Some other thing
1. Not even a thing
2. Big thing
3. Small thing
VS

38. Feedback from customer teams
Methods: Meetings, Spreadsheets
Key Challenges:
Inability to Aggregate
How can you compare between top asks?
How do you merge with direct feedback?
Sales Feedback Success Feedback
1. This thing
2. That thing
3. Some other thing
1. Not even a thing
2. Big thing
3. Small thing
VS

39. Feedback from customer teams
Methods: Meetings, Spreadsheets
Key Challenges:
Inability to Aggregate
How can you compare between top asks?
How do you merge with direct feedback?
Hard to Trust
Second hand reports instead of primary sources.
Sales Feedback Success Feedback
1. This thing
2. That thing
3. Some other thing
1. Not even a thing
2. Big thing
3. Small thing
VS

40. Feedback from customer teams
Methods: Meetings, Spreadsheets
Key Challenges:
Inability to Aggregate
How can you compare between top asks?
How do you merge with direct feedback?
Hard to Trust
Second hand reports instead of primary sources.
Time Consuming & Frustrating
For PMs and customer facing teams.
Sales Feedback Success Feedback
1. This thing
2. That thing
3. Some other thing
1. Not even a thing
2. Big thing
3. Small thing
VS

41. Feedback from customer teams
Method: Custom or 3rd party tool
for logging feedback directly into
system of record.
SOLUTION
- Know originating end-user - Feedback through
teams instead of from.

42. Feedback from customer teams
Method: Custom or 3rd party tool
for logging feedback directly into
system of record.
SOLUTION
- Know originating end-user - Feedback through
teams instead of from.
- Store primary source feedback - What did the
user say in their own words.

43. Feedback from customer teams
Method: Custom or 3rd party tool
for logging feedback directly into
system of record.
SOLUTION
- Know originating end-user - Feedback through
teams instead of from.
- Store primary source feedback - What did the
user say in their own words.
- Conserve limited Product Manager time -
Crowdsource organization to customer facing
teams + Skip the weekly feedback meeting.

44. Putting it All
Together

45. 60%% of customers we have feedback from

46. Now all feedback regardless of channel
can be aggregated in a single system
USER FEEDBACK IDEAS
PM team mines for
problems to solve.
Organized into
ideas by users or
customer team
members.

47. Now all feedback regardless of channel
can be aggregated in a single system
USER FEEDBACK IDEAS
PM team mines for
problems to solve.
Organized into
ideas by users or
customer team
members.
FEATURES
Solutions spec’d by
PM team to
address one or
more ideas

48. Now all feedback regardless of channel
can be aggregated in a single system
USER FEEDBACK IDEAS
PM team mines for
problems to solve.
Organized into
ideas by users or
customer team
members.
FEATURES
Solutions spec’d by
PM team to
address one or
more ideas
CUSTOMERS
CRM data about
customers

49. Takeaways
Build a system of record for all feedback
- Use best practices to get comprehensive feedback both directly and
through customer facing teams. Avoid anecdotes and small numbers.
- Make sure the data structure enables you to combine that feedback.
Avoid death by a thousand spreadsheets.
- Invest early. Cost is lower than you think and the payoff grows over time.
Use a goal-based scoring framework for making
product decisions
- Make sure that decision framework is consistent and understood across
teams. AKA Show Your Work
- Use your database of feedback to de-risk functionality you put on at the
top of the TODO list.

50. Inform & de-risk the product roadmap
Increase internal confidence in the roadmap
Exposing your decision-making framework and the underlying data makes
generating buy from customer facing as well as engineering teams easy.
Focus PM time
Our survey found that PMs spend 20-25% of their time gathering and
organizing feedback. Free them up to focus on higher order functions.
Develop more evangelist customers
People are often blown away when companies close the loop on product
feedback (even if many months later). Don’t discount this aspect of being
customer-centric.
Benefits Thank You!
FOR FURTHER READING
eBook - Build a Customer Feedback Machine
eBook - The Product Manager's Guide to Smart
Product Roadmap Prioritization
webinar - Best Practices for Managing
Customer Feedback at Scale
video - How UserVoice uses UserVoice