[CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol
•
3 likes•5,323 views
Ideas are great - no doubt - but what do you do once you have an entire backlog of ideas? Prioritization is a critical part when building an efficient, impactful testing program. Creating a framework to make smarter choices and thinking deeply about the key factors will make your program successful in the long term. In this session we will talk about how to create a strong prioritization process, how to keep it running, and how to constantly benchmark and optimize your process.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to [CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol
Similar to [CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol (20)
More from CXL
More from CXL (20)
Recently uploaded
Recently uploaded (20)
[CXL Live 16] What to Test Next - Prioritizing Your Tests by Pauline Marol
- 2. Agenda • Where to start • Defining Prioritization Rules • The Hotwire framework • Monitoring the Health of your Optimization Program 2
- 4. Why you need a prioritization process • Managing stakeholders • Bring transparency to optimization • Build trust from the company • Empower employees 4 Make your life easier
- 5. Know your destination • Testing profile • Product strategy • Optimization goals 5 Know what you want
- 6. Know yourself • Get your analytics right • Listen to your customers • Look at your competitors 6 Identify your opportunities
- 7. Structure your ideas • Translate ideas into hypotheses • Centralize • Organize and tag 7 Build your backlog of hypotheses
- 11. Reach Lift Strategic fit Identify the drivers of Value and Effort 11 Value Effort How many people will be impacted? Reasons to expect a strong reaction Aligned with strategy All the frameworks break down value and/or effort into sub-elements, here are the most common. Creative Development Coordination
- 12. In the PIE framework 12 Value Reach Lift Strategic fit Effort Creative Development Coordination Potential Importance Ease Breaking down Value into Potential and Importance
- 13. In the Monetate model 13 Value Reach Lift Strategic fit Effort Creative Development Coordination Potential Creative LOE Technical LOE Breaking down Effort into Creative and Development
- 14. In the Bryan Eisenberg rules 14 Value Reach Lift Strategic fit Effort Creative Development Coordination Impact Resources Time Breaking down Effort into Resources and Time
- 15. Define binary value rules that: • Are objective (yes/no) • Discriminate ideas • Leverage your knowledge Taking it to the next level: The Points model 15 Value Reach Lift Strategic fit Effort Creative Development Coordination Define effort rules that: • Represent your bottlenecks • Are aligned with existing sizing Break down each sub-element into specific rules, to remove personal bias
- 17. 1. Primary metric is conversion 2. Adding or removing info 3. Research/analytics 4. Benchmarking In the Points model 17 Value 1. Above the fold 2. All customers will see it 1. $$ product 2. Cancellation 3. Choice 4. Multiple themes Effort Creative Development Coordination UX/UI points Copywriting needs Engineering points Reach Lift Strategic fit
- 18. Hotwire Car prioritization If we add a Free cancellation flag on the checkout page Then our conversion rate will increase Because customers will feel more confident about reserving a car knowing that they can change their mind 18
- 20. Fold Test 1 Hotwire Car prioritization Free cancellation flag 20 Fold 100% Test 1 0 Fold 100% Main metric Test 1 0 1 Fold 100% Main metric Addingor removing Test 1 0 1 1 Fold 100% Main metric Addingor removing Analytics or research Test 1 0 1 1 0 Fold 100% Main metric Addingor removing Analytics or research Benchmarking Test 1 0 1 1 0 1 Fold 100% Main metric Addingor removing Analytics or research Benchmarking $$ product Test 1 0 1 1 0 1 0 Fold 100% Main metric Addingor removing Analytics or research Benchmarking $$ product Cancellation Test 1 0 1 1 0 1 0 0 Fold 100% Main metric Addingor removing Analytics or research Benchmarking $$ product Cancellation Choice Test 1 0 1 1 0 1 0 0 0 Fold 100% Main metric Addingor removing Analytics or research Benchmarking $$ product Cancellation Choice 2 themes Test 1 0 1 1 0 1 0 0 0 1 Fold 100% Main metric Addingor removing Analytics or research Benchmarking $$ product Cancellation Choice 2 themes Total Test 1 0 1 1 0 1 0 0 0 1 5 Fold 100% Main metric Addingor removing Analytics or research Benchmarking $$ product Cancellation Choice 2 themes Total UX Dev Test 1 0 1 1 0 1 0 0 0 1 5 S S
- 21. Hotwire Car prioritization 21 Fold 100% Main metric Addingor removing Analytics or research Benchmarking $$ product Cancellation Choice 2 themes Total UX Dev Test1 1 0 1 1 0 1 0 0 0 1 5 S S Test2 1 1 1 1 1 0 1 0 0 0 6 M L Test 150 0 0 1 1 1 1 1 1 1 1 8 L S
- 23. Use KPIs to monitor and communicate performance Choose metrics based on your risk profile and testing strategy. We recommend: • Velocity • Win Rate • Conversion lift If your KPIs go down, revisit your rules.
- 24. Benchmark your prioritization rules to stay relevant Validate the correlation between score and reality • Value score vs. Win rate If your rules efficiency flattens, revisit your rules. 0% 25% 50% 75% 100% 0 2 4 6 8 10 Win Rate Value Score Benchmarking Value Points
- 25. Share and Educate • Share your prioritization rules and be transparent • Get feedback to make the rules better • Educate and scale outside your optimization team When you aggregate feedback, revisit your rules.
- 26. Build knowledge • Aggregate results • Translate CR lift into learning and customer insights • Adjust your testing profile, your optimization goals As you accumulate learning, revisit your rules.
- 27. Your turn! 27 1 2 3 Do your research Build prioritization rules Keep improving them