Deep Dive: Testing in an Automated World
•Download as PPTX, PDF•
1 like•4,750 views
Marketers have long struggled with email testing. Deciding when, how, and what to test can get overwhelming very quickly. When you're running automated campaigns on top of that, all but the most dedicated marketers among us tend to throw testing out the window. Don't despair! Marketo's Senior Product Marketing Manager, Matt Zilli, has the answer! In this can't-miss webinar, he will dive into: - Email testing best practices - The latest in testing functionality - Much more!
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Deep Dive: Testing in an Automated World
Similar to Deep Dive: Testing in an Automated World (20)
More from Marketo
More from Marketo (20)
Recently uploaded
Recently uploaded (20)
Deep Dive: Testing in an Automated World
- 1. © 2013 Marketo, Inc. Marketo Proprietary and Confidential Matt Zilli Senior Product Marketing Manager Marketo mzilli@marketo.com @mattzilli
- 2. Page 2 © 2013 Marketo, Inc. Marketo Proprietary and Confidential
- 3. Page 3 © 2013 Marketo, Inc. Marketo Proprietary and Confidential
- 4. Agenda 1 2 3 Intro to A/B Testing Best & Worst Practices A/B Testing in Marketo
- 5. Page 5 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Why aren’t more marketers actively testing?
- 6. Page 6 © 2013 Marketo, Inc. Marketo Proprietary and Confidential My favorite excuses 1. I have no time for this 2. The results are rarely conclusive 3. It’s too hard! 4. I already know the answer
- 7. Page 7 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Which Subject Won? Test email subjects from the registration email for the 2014 Marketing Sherpa Email Summit: Your emails don’t work (We’ll tell you why) What happens in Vegas…will improve your emails! Save $300 now. (Open BEFORE Christmas) Email Summit 2014: discount ends Jan 9th Sonia Simone, Copyblogger, Jan 2014
- 8. Page 8 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Which Subject Won? Test email subjects from the registration email for the 2014 Marketing Sherpa Email Summit: Your emails don’t work (We’ll tell you why) Middle What happens in Vegas…will improve your emails! Save $300 now. Loser (Open BEFORE Christmas) Email Summit 2014: discount ends Jan 9th Winner Sonia Simone, Copyblogger, Jan 2014
- 9. Page 9 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Things to test: Landing Pages
- 10. Page 10 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Things to test: Website
- 11. Page 11 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Things to test: Emails From name/address Subjects Whole Emails Date/Time
- 12. Page 12 © 2013 Marketo, Inc. Marketo Proprietary and Confidential 7% Better than B 30% Better than B The loser A. B. C. Jason Fried, signalvnoise.com, Jan 2009 Which headline won?
- 13. Page 13 © 2013 Marketo, Inc. Marketo Proprietary and Confidential 7% Better than B 30% Better than B The loser A. B. C. Jason Fried, signalvnoise.com, Jan 2009 Which headline won?
- 14. Page 14 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Testing Best Practices
- 15. Page 15 © 2013 Marketo, Inc. Marketo Proprietary and Confidential #1: Send to a large sample size, test a single variable
- 16. Page 16 © 2013 Marketo, Inc. Marketo Proprietary and Confidential
- 17. Page 17 © 2013 Marketo, Inc. Marketo Proprietary and Confidential #2: Check for statistical significance Calculator: www.zettasphere.com/abcalculator
- 18. Page 18 © 2013 Marketo, Inc. Marketo Proprietary and Confidential #3: Over time, test everything Subject lines, from names/address, content, text, templates, CTAs, creative, button colors, link order….
- 19. Page 19 © 2013 Marketo, Inc. Marketo Proprietary and Confidential #4: Whenever you can, measure conversions, not opens/clicks
- 21. 21% Better dmix.ca Blog, 2011
- 22. Page 22 © 2013 Marketo, Inc. Marketo Proprietary and Confidential New things • Test Deeper CTAs Chris Hexton, MarketingLand
- 23. Page 23 © 2013 Marketo, Inc. Marketo Proprietary and Confidential New things • Test “Mad Libs” Forms – 25-40% Increases Luke Wroblewski, LukeW Ideation & Design, Feb 2010
- 24. Page 24 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Things to Avoid: 1) Relying on test results that are insignificant 2) Missing opportunities to test & learn 3) Unstructured tests
- 25. Page 25 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Miles Gotcher, Marketo Blog, Oct. 2013
- 26. Page 26 © 2013 Marketo, Inc. Marketo Proprietary and Confidential 12% Higher Click-to-Open Miles Gotcher, Marketo Blog, Oct. 2013
- 27. Page 27 © 2013 Marketo, Inc. Marketo Proprietary and Confidential A/B Testing in Marketo
- 28. Page 28 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Batch Email A/B Test
- 29. Page 29 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Batch Email A/B Test
- 30. Page 30 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Batch Email A/B Test
- 31. Page 31 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Batch Email A/B Test
- 32. Page 32 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Batch Email A/B Test Fills out Form Visits web page Forwarded to a Friend Shared on Social Networks
- 33. Page 33 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Batch Email A/B Test
- 34. Page 34 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Testing in Triggered & Engagement Campaigns
- 35. Page 35 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Testing in Triggered & Engagement Campaigns
- 36. Page 36 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Testing in Triggered & Engagement Campaigns
- 37. Page 37 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Landing Page A/B Test
- 38. Page 38 © 2013 Marketo, Inc. Marketo Proprietary and Confidential Takeaways 1. Test whenever you can 2. Check for statistical significance 3. Go beyond batch tests 4. Test whole funnel, not just opens/clicks
Editor's Notes
- Email tests – This story was covered in an interview by eConsultancy The Obama campaign team carried out A/B tests on 15%-20% of its email list, which included around 15m people in total. Typically the tests involved four different messages that were each split out by three different subject lines. The messages would range from focusing on looming election deadlines to personal pleas from Obama. Some of the tests achieved an additional $2.2m in revenue from a single email by sending the best draft compared to the worst draft. After finding the best subject line and draft Showalter’s team began testing other aspects of the email design, like the unsubscribe language and personalised content. Personalised content included the greeting and the amount to ask for based on each recipient’s previous donation history. The team added ‘drop in’ sentences to target those who had already contributed, such as “thanks for donating last time, please dig deep to donate again.” This type of personalisation significantly raised donation rates. This was kind of revolutionary at the time and overall more than half of the $1bn raised by the campaign came from online donations. This A/B testing probably resulted in $200m in additional revenue. The emails helped to gain donations from more than 4m Americans, recruited tens of thousands of volunteers and publicised events. Takeaways: 1) It can have a huge impact; 2) It can be done over time; and 3)….
- 3) It’s an opportunity to learn; The more you know about your prospects, consumers and customers, the better your marketing will be and the more competitive your company will be
- A quick primer, the good & the bad, including some cool things I came across, and a quick dive into the product Intro to A/B Testing What can I test? Best practices & Worst practices Marketo Marketing Automation Features / Demo Landing Pages Batch Emails Trigger / Engagement Emails
- Raise your hands if this applies to you
- You know what happens when you take shortcuts…
- A quick test – You must participate! These subjects were all tested in an email that was trying to drive registration for the Marketing Sherpa Email Summit. Which do you think won?
- A quick test – You must participate! These subjects were all tested in an email that was trying to drive registration for the Marketing Sherpa Email Summit. Which do you think won?
- So where should you be thinking about testing? How many people are testing landing pages today?
- This is an example of testing your website. There are tons of tools out there, like Optimizely, which is integrated with Marketo. How many people are running AB tests on their website? This example here is actually using Marketo Real-time personalization for AB testing. In RTP, you can test different versions with different audiences, like industries or specific companies.
- And of course, emails, which is what we’ll mostly talk about today. How many people are running AB tests on their emails today?
- The #1 goal of testing. But notice I said goal, not RULE. I believe you can learn a lot of small tests. But for the really important tests, this is the right goal. Because if you do this, you can be very confident in the results. Why?
- Well, we’re going to walk through this math step-by-step because who doesn’t love doing math? Seriously though, AB testing relies on statistics. And that means that when you run a test, you want a result like “I’m 95% confident that these results represent reality” When you send to a large sample, the likelihood of that goes up. But with smaller samples, you may only be able to say “I’m 50% confident…” And then what’s the point? Also, if you try to test multiple things, did you really learn anything? You learned which email will work this one time, but you won’t really know which change caused the different results.
- So, to that end – if you really want to know whether your results mattered, plug them into a calculator. There’s a link here you can check out, it’s really not hard.
- Third – NEVER MISS AN OPPORTUNITY TO LEARN SOMETHING. As we’ll see in the product, AB testing is easy enough now to just tack it on to any email you send and learn something new about your audience.. Going back to how I started – the more you know about your recipient/visitor/prospect/customer/etc….
- Last but not least, AB test on things that matter. It’s true that email opens are a good indicator of good subjects, but even that’s going away with mobile pre-headers. To really understand the effectiveness of an email, measure whether the action was completed.
- Who knows why? But that’s the point of these tests!
- Who knows why? But that’s the point of these tests!
- Just a couple of new things I wanted to include – Amazon tests whether a single big call to action beats deeper links into a site.
- This company got conversion increases of 25-40%
- 3) Have a hypothesis and know what you’re testing
- Product demo Batch testing Custom Conversion Testing in a trigger campaign Testing in an engagement stream Custom Conversion uses Fills out form Visits web page Forwarded to a Friend Shared on Social Networks