Emails have been around since the time of dinosaurs (maybe just the late 70s, but you get the point) - yet, nobody has ever sent the perfect email. While the perfect email might not even exist, how can you begin to create better emails with each and every send? Watch Tim Ozmina, Sr. Marketing Specialist, for his webinar, Acing the A/B Test: How to Optimize to Send Extraordinary Emails, where he dives into the world of email A/B testing. You'll learn: -What you should A/B test in your emails -How to set up a proper A/B test -How to accurately report on your results and make decisions

1. T I M O Z M I N A
Acing the A/B Test: How to Optimize to Send Extraordinary
Emails

2. 2Proprietary and Confidential | © Marketo, Inc.
• This webinar is being recorded, the slides and recording will be sent to you once the webinar
con clu des
• Qu estion s? Feel free to en ter th em in to th e Q&A box on y ou r screen an d we’ll get to th em at th e
en d of th e session
• Postin g to social? Please u se ou r h ash tag # m ktgn ation
Housekeeping

3. Proprietary and Confidential | © Marketo, Inc.
Meet the Speaker
Tim Ozmina
Sr. Marketing Specialist,
Demand Generation

4. 4Proprietary and Confidential | © Marketo, Inc.
A Brief History of Emails
 1971 –Firstemail was sent
 1985 –Email is usedby government workers,military,
academics
 1991 –Thefirstemail wassent fromspace
 1996 –Microsoftand Hotmail releaseemail tools
 1997 –Microsoftbuys Hotmail; almost 10 million
emailusers
 1998 –Theterm“Spam” enters the dictionary
 2003 –Over77 millionusers
 2004 –U.S. Governmentstartsregulatingunsolicited
emails
 2009 –94% ofallemails werespam
 2019 –3.9 billionemailusers; 293 billionemails sent
per day; 64% ofemailis spam
Sources: MacWorld, 99firms, Radicati, EMarketer

5. 5Proprietary and Confidential | © Marketo, Inc.
Why Should You A/B Test?
What if you don’t test?
 Youdon’tlearnwhatis workingandwhatisn’t
 Youmissouton meaningfulengagements
 Yourgraphsdon’ttrendup andto theright
 Lossofrevenue
 Youdon’tgrow
Whatifyou dotest?
 You canbreakthroughthenoise of 293 billion emails sentper day
 You canoptimize EVERYTHING
 You canlet thedatado thetalking
 You captureMORE meaningful engagements
 Increaserevenue
 You grow

6. 6Proprietary and Confidential | © Marketo, Inc.
Now We Know Why… But WhatIs the Process?
Define Your
Goal
Decide
What to
Test
Create a
Hypothesis
Test
Evaluate
Results
Make a
Decision

7. 7Proprietary and Confidential | © Marketo, Inc.
It’s Time to Pick a Test!
What should you test?
 Subject line
 Numberofwordsin SL
 Personalizedvs. Non-personalized
 Brackets
 Call-to-actionbutton
 Size
 Color
 Font
 Italicizevs. Bold
 Preheader
 Tone of voice
 HTML vs. Text email
 Day of week
 Time of day
 Length of email
 From name
 And many, many more!

8. 8Proprietary and Confidential | © Marketo, Inc.
How Do You ActuallyRunthe Test?
A few things to keep in mind:
 Onlytestone element atatime
 Keep thetesting environment asclean aspossible
 Send tothe same audience
 Send at the same time
 Gatherasmuch dataaspossible
VS

9. 9Proprietary and Confidential | © Marketo, Inc.
Before We Evaluate…
Statistical Significance:
 anumberthatexpressestheprobabilitythattheresult ofagiven experimentorstudycouldhaveoccurredpurelybychance. Thisnumbercan
beamargin oferror(“Theresults ofthispublic opinionpoll areaccuratetofivepercent”), oritcan indicateaconfidencelevel (“If this
experimentwererepeated,thereis aprobabilityofninety-fivepercentthatourconclusionswould besubstantiated”).

10. 10Proprietary and Confidential | © Marketo, Inc.
Now It’s Time to Evaluate
Results:
 Open ratesincreased by4% (100% statisticalsignificance)
 Click-through-ratesincreasedby47% (100%statisticalsignificance)
 Click-to-openratesincreasedby41% (100%statisticalsignificance)
(ThanksMike!)
What doesthis mean?
 Contextis amazing…
 Adding contextnotonlyincreased opens,butalsoincreased clicks
(ThanksMike!)
Specifically for this test:
 Earlystageaudience
 Marketingtitles
 Engagedin thepast1year

11. 11Proprietary and Confidential | © Marketo, Inc.
And theWinner Is…
WhatdoI donow?
 DOCUMENT YOUR FINDINGS!
 Letyour teamknow
 Include bracketsto thataudience
 Testthison your otheraudiences (if you weren’ttestingitalready)
 Pick anothertest
 Takeit astepfurther

12. 12Proprietary and Confidential | © Marketo, Inc.
Let’s Take It a Step Further

13. 13Proprietary and Confidential | © Marketo, Inc.
I KnowWhat You’re Thinking:
We just beat the control and now knowthat adding
brackets to the subject line can help with not only
opens,but also clicks… what else is there?

14. 14Proprietary and Confidential | © Marketo, Inc.
Always Be Testing
VS

15. 15Proprietary and Confidential | © Marketo, Inc.
The ResultsLook Conclusive
Results:
 Open ratesincreased byover 15% (100%statisticalsignificance)
 Click-through-ratesincreasedbyalmost30% (100% statisticalsignificance)
 Click-to-openratesincreasedbyover 12% (100% statisticalsignificance)
 Looksconclusive, right?

16. 16Proprietary and Confidential | © Marketo, Inc.
The FullStory

17. 17Proprietary and Confidential | © Marketo, Inc.
Time to Break a Myth

18. 18Proprietary and Confidential | © Marketo, Inc.
Adding a Secondary Call-to-Action
 We’vebeentoldit’sa no-no
 Decreasesclicks
 Increasesunsubscribes
 Can hurtdeliverability
 Have you ever testedit?
 For thistest:
 Senttoallbehaviorscoreaudiences
 Marketingtitles
 Over 450kemails
 Engagedinpastoneyear

19. 19Proprietary and Confidential | © Marketo, Inc.
The Results
Results:
 Open ratesincreased by0% (no statisticalsignificance)
 Click-through-ratesdecreasedby3% (no statisticalsignificance)
 Click-to-openratesdecreasedby3% (no statisticalsignificance)
What does this mean?
 Wecannotconfidentlysaythataddinga secondCTA directlyincreases/decreases the
performanceofan email
 Disclaimer:Trythisonyouraudience, don’tjust takemy wordforit!

20. 20Proprietary and Confidential | © Marketo, Inc.
5 Key Takeaways
 Documentyour findings
 Keepyour tests clean
 Dig deeper
 Find statisticalsignificance
 Always be testing

21. 21Proprietary and Confidential | © Marketo, Inc.
Questions?
Tim Ozmina: https://www.linkedin.com/in/timothy-ozmina/