Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Testing Your Testing Program

What does digital marketing maturity look like? How can companies effectively benchmark their experimentation performance? Brooks Bell will share a proven framework that allows you to unlock experimentation success, make more efficient investments and confidently plan for growth.
You will learn:
-the six elements of a successful experimentation program
-how to benchmark your performance
-proven ways to evolve your digital marketing maturity

  • Login to see the comments

Testing Your Testing Program

  1. 1. Testing Your Testing Program Presenters Claire Schmitt, Sr. Director Optimization Consulting, Brooks Bell David DeFranza, Director, Optimization Consulting, Brooks Bell Hudson Arnold, Strategic Partnerships, Optimizely
  2. 2. Today’s Speakers Claire Schmitt Senior Director Optimization Consulting Brooks Bell David DeFranza Director, Optimization Consulting Brooks Bell
  3. 3. Housekeeping • We are recording • Slides will be available after the webinar is complete • There will be time to submit questions at the end of the presentation
  4. 4. Agenda • Overview • The Brooks Bell Maturity Model • Key questions to ask about your own program • Testing program problems + solutions • Q&A
  5. 5. Brooks Bell is a premier experimentation consultancy offering world-class brands a flexible, consulting-based operational model. Our team of testing experts create custom, data-driven strategies that help companies solve problems facing their business. Who we are Program Advisement Experimentation Training Full Service Testing Resource Augmentation
  6. 6. Building a testing program is hard Testing requires a lot of time and attention. As a result, testing managers tend to focus on daily tasks and not overall program performance.
  7. 7. We get tests out the door, but... • Are we generating impact consistently? • Does each test contribute to a growing body of knowledge? • Is our process as efficient as it could be? • Are we consistently increasing the skills of our team? • Has participation across the organization increased? • Does data drive our decision-making?
  8. 8. The Brooks Bell Maturity Model
  9. 9. Digital Experimentation Maturity Model 6 Critical Testing Program Pillars Culture Team Process Tech & Tools Strategy Performance 6 Key Program Pillars
  10. 10. Culture Team Process Tech & Tools Strategy Performance 6 Key Program Pillars Culture Categories • Demand • Buy-in • Motivation • Testing resources/support • Levels of engagement • Training/education Team Categories • Testing organization structure • Testing roles/responsibilities • Skillset • Resource availability • Executive sponsorship Team ProcessProcess Tech & Tools Strategy Performance Team Culture Process Categories • Experiment design • Execution process • Documentation • Communication • Reporting + sharing • Standardization • Test coordination • Queue prioritization Team Process Tech & Tools ProcessProcess Tech + Tools Categories • Tool stack • Data quality • Integration with existing systems • Tool feature/functionality • Tool utilization Tech & Tools Strategy Strategy Categories • Methodology • Data usage • Testing locations • Ideation maturity • Customer centricity • Journey • Iterative testing Performance Strategy Performance Categories • Goals • % capacity testing • Program measurement • Impact measurement • Results/Insights tracking
  11. 11. Digital Experimentation Maturity Landscape 1 2 3 4 5 LEVEL 1 • Testing program in infancy • Goals may not be defined • Testing org structure not established yet • No processes or standardization • Strategy not informed by data • Testing and analytics tools in place LEVEL 2 • Testing is generally accepted • Goals and metrics established • Org Structure in place, testing skills inconsistent • Basic processes in place • Strategy inconsistently informed by data • Testing and analytics tools integrated LEVEL 3 • People want to test and community is forming • Program focused on maximizing opportunities • Resource(s) are dedicated to testing • Standard, comprehensively managed process • All strategies informed by data • Some features of tools are being maximized LEVEL 4 • Testing informs majority of site decisions • Testing focused on impact • Cross-functional, experienced resources • Processes adopted and optimized for efficiency • All strategies informed by qual and quant data • Utilization of tool features maximized LEVEL 5 • Testing and data is innate • Testing impact and opportunities maximized • All individuals are versed in all aspects of testing • Processes is flexible for rapid deployment • Comprehensive consumer behavior strategy • Utilizing advanced features within tool stack
  12. 12. Level 0 1 2 3 4 5 Getting to a level 1 is a true sign of success
  13. 13. Evaluating your own testing program
  14. 14. We need to stop and ask ourselves... STRATEGY • Is there a standard process for generating test ideas? • Are test ideas typically informed by data? • Is the ideation process is widely adopted and consistently applied? TEAM • Is there basic knowledge of and experience with testing? • Are resources at least partially dedicated to testing? • Is an experienced, multi-disciplinary testing team is in place? PROCESS • Has a test execution process been defined? • Is the test execution process comprehensively managed? • Is a standard process documented thoroughly and widely distributed? TECH & TOOLS • Has a testing tool been implemented? • Is the testing tool fully integrated with an analytics platform? • Are features of the experimentation stack maximized? CULTURE • Is there acceptance of testing as a decision-making tool? • Is there a demand for testing and data? • Is testing innate and embraced at every level of the organization? PERFORMANCE • Are some tests being launched? • Are goals and success metrics for the program established? • Is test impact consistently measured?
  15. 15. Increasing your testing maturity • Accelerating testing output • Maximizing KPI impact • Improving customer experiences • Increasing data informed decisions • Scaling you program As you build and execute your testing program – focus on: Culture Team Process Tech & Tools Strategy Performance 6 Key Program Pillars
  16. 16. There are some common problems we see with each of the 6 pillars.
  17. 17. Problems with Culture • Teams do not understand what testing is or why they should test • Lack of motivation or excitement to test • No training or foundation in testing • Lack of community resources to support testing efforts • Limited or no engagement by Sr. Management Culture
  18. 18. What to do first Hold a Testing Summit Bring stakeholders together for a full or half-day event that includes case study reviews, training sessions, ideation roundtables, and guest speakers. Though it can take a lot of planning and coordination, such an event offers a valuable opportunity to kick start a stuck culture. Culture
  19. 19. Team Problems with Team • Organizations do not have a Center of Excellence or the right testing program structure • Roles/responsibilities are not clear • Not the right or not enough resources • Lack of skills • Not enough resources
  20. 20. TeamWhat to do first Institute a Training Program It would be great to supercharge an inexperienced team with fresh talent, but doing so is rarely practical. Instead, institute a regular training program—which could be as simple as weekly “brown bag” lunch programs or a regular reading group—to build skills among the team already available.
  21. 21. Process Problems with Process • Inefficient process for execution/lack of standardization • Communication challenges • Lacks clear and consistent documentation • Experiment design lacks rigor • Insufficient prioritization of tests • Results aren’t being shared
  22. 22. What to do first Document the Process If your process feels stuck or broken, it’s time to capture it and create a formal outline—or conduct a thorough review if such an outline already exists. The outcome should be a detailed RACI that defines roles and responsibilities from initial ideation to the final implementation of winning treatments. Process
  23. 23. Problems with Tech & Tools • Testing tools and analytics tools aren’t the right fit for organization needs • Tools are not integrated • Lack of tool capabilities • Individuals to not know how to use the tools • Utilization of tools is low and limited Tech & Tools
  24. 24. Tech & ToolsWhat to do first Audit the Tech Stack Though tech challenges can be the most costly and challenging to fix, there are usually simple improvements to improve the performance. Begin by outlining the stack and its components, looking for errors in integration, discrepancies in data, and missing metrics of interest. Once this information is compiled, a roadmap of fixes can be prioritized
  25. 25. Strategy Problems with Strategy • Strategy maturity is low • No process for developing testing ideas • Lack or limited use of data to support ideation • Teams lean heavily on either qualitative or quantitative data – often times not both • No roadmaps or not prioritized based on key considerations
  26. 26. What to do first Adopt an Ideation Methodology Weak test strategies typically stem from a lack of formal process. Instituting a methodology ensures data is used to inform ideas, reduces the number of opinion-based tests, and facilitates prioritization of test queues. Strategy
  27. 27. Problems with Performance • No testing goals for organization or self- service teams • Program performance across the organization is not being captured or monitored • Testing impact not being measured or consistently measured • Testing capacity not being maximized • Testing results/insights not being surfaced or used Performance
  28. 28. PerformanceWhat to do first Define Goals for Testing Formal goals for the testing program help to inform operational strategies, drive improvement and, perhaps most importantly, communicate the role and mission of the testing team across the organization. If you’re testing program is floundering, define a set of specific goals for the program right away.
  29. 29. Action Plan 1. Hold a testing summit 2. Institute a training program 3. Document the process 4. Audit the tech stack 5. Adopt an ideation methodology 6. Define goals for testing Culture Team Process Tech & Tools Strategy Performance 6 Key Program Pillars
  30. 30. Key Takeaways 1. We need to consistently take a 10,000 foot view of our testing programs 2. Tracking our growth across six pillars helps focus the effort 3. Identifying challenges and weaknesses within each pillar allows us to create effective growth strategies
  31. 31. Questions? Please submit your questions via the text box on your screen.
  32. 32. 1200+Attendees 20+Sessions 100+Giveaways 30+Speakers Wynn Las Vegas, October 17-19 Registration Early Bird $750Expires on 09/01/2017 Early Early Bird $550Expires on 07/01/2017 Regular $850Expires on 10/16/2017
  33. 33. Thank you