The document summarizes the results of various A/B tests Marketo conducted to optimize their marketing campaigns. Some of the key tests discussed include:
1. Testing different email send times, finding that emails sent at 1pm performed best with higher open and click-through rates.
2. Testing personalized vs generic emails, finding that generic performed better for early stage leads but personalization was better for late stage leads.
3. Testing HTML vs text emails, finding that text emails surprisingly performed better with higher click-through rates.
4. Testing webinar optimization for different lead stages, finding early stage leads registered but mid/late stage leads attended at higher rates.
5. Testing different
A/B testing, also known as split testing, compares two versions of the same campaign on a certain channel. These channels include (but are not limited to):
Website
Social Media
Emails
Mobile Apps
Landing Pages
Digital Ads
For example, you could set up a subject line test for adding in personalization (i.e. the person’s first name). Your company hasn’t done that before, but you hope to find that adding in that element will increase overall email engagement. After running the test, you find that adding in the person’s first name increases open rates by 10%, does nothing to click to open rates, but improves conversions on the landing page by 50%.
As in this example, A/B testing is about testing only one element at a time so that marketers can maintain control over the results and draw firm conclusions. From these conclusions, marketers are able to assess the effectiveness of each variable in driving conversions and from there, can make the necessary adjustments to their campaign. By making these necessary adjustments, marketers are able to continue improving their efforts in order to deliver the ROI possible to their organization.
So why is A/B testing important? By assessing the actions of buyers, A/B testing reveals what truly appeals to them. In doing so, it also advances engagement, campaign effectiveness, and marketer expertise. Here are a few specific reasons why A/B testing should be a part of your company’s marketing strategy:
A/B testing increases engagement with your buyers
A/B testing enhances campaign effectiveness and optimizes programs for a company’s target audience
A/B testing enhances a marketer’s awareness and expertise about audience preferences
So let’s say you’re ready to run an a/b test. How should you go about setting it up? Here’s the process I follow:
Choose what to test – think about testing elements that you know would be high impact. Something like a sign-up page, welcome email, or pricing page would be a great place to start.
Ask a question and then write a hypothesis – just like an scientific method, a/b testing begins with developing a hypothesis to answer a question. You can base your hypothesis on what has worked on other digital assets, insights from your colleagues or just plain old instinct!
Decide on the sample group – who are they, what are their past behaviors, and how large is the sample size are critical details. Make sure your test segment is big enough to show results!
Define what success looks like – here, you are trying to figure out what you’re trying to achieve through testing. What are your ultimate success metrics? What will you improve through testing and optimization?
Set up the test – this one is straightforward enough.
Look at the test results – use the success metrics you decided on earlier to evaluate your results. We will talk about this more in a moment.
Determine the winning combination – which version performed better? Did it perform statistically better or just marginally? Was it statistically significant?
Implement the necessary changes – roll out the changes, educate your team, create a process, and start working on the next test!
For all of my tests, I use this simple a/b testing calculator by Kissmetrics! In the first column, you enter your total number of observations, which means if you’re doing landing page testing, this would be total page views. If you were doing subject line testing, this would be emails delivered.
In the second column, it’s the total number of conversions. This is the actual success number.
Between these two values, it gives you a conversion rate and a confidence interval for significance. Normally, you’ll find that calculators are set up for a 95% confidence interval, which means that in order for you to see statistical significance, you need to be at least 95% certain that your test results improve the conversion rate.
If you didn’t use a calculator like this, you might be making changes that aren’t actually statistically significant, which can hurt your conversion rates and the ROI on campaigns.
UNCLEAR SUBJECT LINES
Here is the question we are trying to answer: What is the best time of day to send an email?
Details:
Split total audience 3 ways
Backed into numbers to see at least 1,000 opens per test (for sig.)
Sent each email from a different campaign to ensure exact timing on the send
Tested the time testing over 7 different email campaigns
Over 200,000 emails sent
Tested in early, mid, and late buying stages
Results by significance:
1pm has a 14% higher open rate than 5am (100% statistical significance)
1pm has a 5% higher click to open rate than 5am (statistically insignificant)
1pm has a 20% higher click through rate than 5am (100% statistical significance)
1pm has a 7% higher unsubscribe rate than 5am (statistically insignificant)
1pm gives us 20% higher email engagement
What the industry says about personalization?
Personalized emails deliver 6x higher transaction rates. – Experian
Personalized email messages improve click-through rates by an average of 14% and conversions by 10%. - Aberdeen
When asked to prioritize one capability that will be most important to marketing in the future, 33% of marketers answered: “personalization.” - AdAge
So with all these quotes, you’d suspect that any element of an email you can personalize, you should, right?
Well, here is how personalization works within Marketo and how we would normally personalize our emails.
The top section here is a screenshot from the email editor. In the From Name box, you’ll see bracket text. Those are called tokens, which dynamically push in values pulled from fields. The tokens in the From Name send from the Lead Owner. If that email subscriber doesn’t have a lead owner, it will default to Team Marketo.
Same principles apply to the From Address and Reply-to sections where if the email subscriber has a lead owner, that email will come from the lead owner’s email. If they don’t have a lead owner, it defaults to marketoteam@marketo.com.
All emails were sent under these rules until…we tested it! What we found was extremely interesting.
The question we asked ourselves was “Is a personalized email really better?” Is it better under all conditions?
So we ran a simple personalization test. Our control email used our standard personalization using the tokens. If the email subscriber had a lead owner, it would be personalized from them in the From Name, From Address, Reply to address, and email signature.
For the test, we removed all of the dynamic tokens and populated the From Name, From Address, Reply-to and email signature with the default values. So everything came from Team Marketo instead of say, Mike Madden.
The email subject lines and body copy all stayed the same otherwise.
Here are the testing details. We ran a total of 16 campaigns, so 32 unique emails. About 107,000 emails were sent and we testing this lack of personalization to email subscribers in the early, mid, and late buying stages.
Let’s look at the results!
So the way we define early, mid, and late stage email subscribers is by their behavior score. For us, we know the best leads for our sales teams are the ones that have marketing titles and come from the right companies, so we actually hold those scores constant, meaning that you can’t even become a marketing qualified lead if you aren’t the right job title from a good account.
That being said, our buying stages are defined by behavior scores. So for example, someone that has only clicked one email or visited a few web pages would be classified as an early stage email subscriber. As they engage more and more with our digital assets, they would progress into the mid and late buying stages.
For the early buying stage, we found that keep emails generic produced much better results.
We found generic emails to have a:
10% higher open rate
26% higher click to open rate
38% higher click through rate
And all of these numbers have 100% statistical significance! When we step back, this makes a lot of sense. If a new name comes into your database, it probably isn’t a great idea to immediately start sending them emails from their dedicated sales rep. They want to hear from your brand in the early stages, not a sales person. I believe that this helps introduce new names to the brand, softens our messaging a little bit, and helps progress them forward faster into the next buying stages.
For the mid buying stage, I was pleasantly surprised again.
We found generic emails to have:
A 3% lower open rate, which is statistically insignificant but…
A 30% higher click to open rate
And a 26% higher click through rate
We saw 100% statistical significance again in this stage, which again means mid-stage email subscribers aren’t ready to hear from a sales person yet. And we certainly don’t want to rush someone into that relationship because as we all know, people hate being sold to. If you are trying to sell to your buyers too soon, they might get turned off and go with another brand.
Finally, in the late stage, we couldn’t produce any results with statistical significance even after multiple email sends.
We found generic emails to have a:
4% higher open rate
5% higher click to open rate
9% higher click through rate
Now whether or not we saw some directional numbers, this is the part where we need to use our better judgment as marketers. Late stage emails should come from the lead owner because we are, after all, trying to sell software. So for this stage, we kept our normal process for sending emails with personalized tokens.
After all this personalized vs. generic testing, we’ve learned something valuable. Introducing the sales rep too soon is actually worse for our email performance. We are turning off buyers before they can even get to the late stage. And we really need to focus on is simply moving people from left to right with as much email engagement as possible because we know that if we can get email subscribers to engage heavily with our brand (which drives up their behavior score), there’s a really good chance sales people will be able to set up a phone call.
Introducing personalization in the beginning of a buyer’s journey might be worse for the buyer.
Make the right business decision even if a test says otherwise.
Use this test to find the right handoff point for personalization
FAILURE TO MEASURE EMAIL INBOXING
At first glance, the email above seems to have a clear CTA, minimal distractions, and purpose-built layout for a single offer. But if we challenge ourselves to look past this visually pleasing layout, here’s what we see:
5 different links (Marketo logo, 3 social network buttons, and 1 offer link on the banner, copy, and CTA)
A 564×355 pixel header banner (this means it takes at least 355 pixels before the recipient begins reading)
A 564×65 pixel footer banner
A 564 pixel width layout, which confines our copy to a set width, pushing it down into more lines of text
While many of these features help to accommodate key elements for visual branding and different device widths, they also can prevent the subscriber from focusing on the most important element: the main CTA or offer. So now we ask ourselves the hard question. Does including/hyperlinking elements like our company logo and social buttons hurt email performance? Are we losing out on clicks for the main offer?
So here is what our test looked like. The HTML email is visually appealing (because hey, we’re marketers) and the text email is simple and straightforward. Let’s peek at the data.
Over the course of 5 different email drops, we have found text emails to do the following:
Roughly the same open rate as HTML emails
11% higher click to open rates (95% statistical significance)
8% higher click through rates (86% statistical significance)
Now, those stats aren’t a confidence booster! 95% statistical significance for click to open rates is at the low end of the spectrum, meaning I wouldn’t be confident calling it a better email. The click through rates definitely need more testing. But let’s dive deeper into the data.
If we move beyond aggregated clicks for the HTML email and look at individual link performance, we find that not every email click goes to the offer. In fact, nearly 16% of clicks went to other links like social buttons. When the purpose of our email is to promote a specific offer, we want clicks on the offer, right?
When we rerun the numbers solely looking at unique offer clicks, the data strongly favors text-based emails. Text emails show the following performance:
Roughly the same open rate as HTML emails
21% higher unique click to open rates on the offer link (100% statistical significance)
17% higher unique click through rates on the offer link (99% statistical significance)
In the end, we found that focusing a subscriber on a single-link, text based email produced higher clicks on the call-to-action. Sure, a little bit of my soul cried knowing that all this time, the pretty emails weren’t as good as a simple text email. But isn’t that what marketing is all about? We must constantly challenge our beliefs to find the absolute best process for every campaign we run. And if we let our opinions get in the way, we miss out on easy wins just like this one.
Sometimes the pretty emails aren’t the winners.
On the bright-side, text-based emails require less creative resources!
Test these out across asset types. You may find that HTML emails work better for some programs. For us, it is webinars!
NO ENGAGEMENT SEGMENTATION
So like we’ve talked about in this webinar so far, we like to think about our buyers in terms of stages and those stages are defined by behavior scores.
Early stage buyers have a behavior score of 0-5.
Mid stage is 6-19.
And late stage is anything greater than 19.
In order for buyers to progress their score, they need to show enough digital engagement, which would be showing forms of activity like email clicks, form completions, page visits, offer downloads, etc. If a buyer stops engaging, their score will slowly decay back to zero. In order to really progress through the buyer stages, you need to perform multiple actions and remain engaged.
When we think about a database broken down by behavior scores and buying stages, we find about 10% in late stage, 30% in mid stage, and 60% in late stage. Our main goal is progress buyers into the late stage and the secondary goal is to keep them there. Since the score decays, they need to continue their engagements to stay in the late stage.
Knowing the ground rules, we asked ourselves a big question: Who registers for our webinars? Who attends them? Is it the people we expected to join?
At the time I asked this question, I was conveniently speaking on a webinar for Marketo. It was what we considered to be an early stage webinar called “3 Hacks to Boost Email Open Rates”.
We assume this is an early stage topic because it is merely speaking about email marketing in general. We don’t talk about marketing automation or the Marketo product, which we would consider to be much more mid and late stage topics.
So the big assumption here would be that the folks that register for such a webinar are people that fall into the early buying stage.
So after the first email invite, I looked at the registration list and broke it down by buying stages. Here is what I found:
Nearly ALL, I repeat, ALL of the people registered for this quote on quote early stage webinar came from our late buying stage. Seems rather odd, right? Well, it goes against everything we believed to be true for these types of webinars.
And to only see 5 people registered from the early stage was shocking.
So that was weird enough to say, let’s look at another one. So a few months before, we had run the 8 Biggest Mistakes Social Media Marketers Make and How to Avoid Them webinar, which also seems like a very early stage topic. Again, this did not talk about marketing automation or Marketo in any way.
What we found was similar.
The HUGE majority of the people who had registered for this webinar were from the late buying stage. What was really peculiar was if our late buying stage is only let’s say 10,000 people, a really big chuck on them tend to attend our webinars.
But what happens to once the webinar airs? Which types of people attend the webinar?
Well, no surprise here. The late stage people are the ones that attend webinars too. So at this point, I’m thoroughly confused. Everything I thought I knew about this programs was wrong.
It was like it didn’t matter if the topic was early, mid or late stage, people in the late buying stage were going to register and attend. Period.
When we look at the attendance rates, late stage people were more than 2x more like to attend a webinar than those in the early stage. And mid stage folks were pretty close up there with the late stage folks.
So after having my earth shattered a little bit, I thought it would be interesting to go back and revisit the email data from the first webinar invite for the 3 Hacks to Boost Open Rates webinar. I figured if I could break out email performance by buying stage, I might get more insight into what is actually driving these numbers.
In the purple table, you’ll find the raw email performance numbers. In the light yellow tables underneath, you’ll find indexed numbers off of the values above them.
For example, mid/late stage folks had a 131% higher open rate than early stage folks. They also had a 2,320% higher click through rate and a 984% higher click to open rate. Yes, those are the real numbers and they are no joke!
The unsubscribe rates were nearly half. And last but not least, after sending 96,955 emails to early stage folks, we literally only had one registration. One!!!!!!
Let’s just let that sink in for a moment.
Now after a couple more email invites where we excluded the early stage folks, we analyzed the attendee data. We found mid and late stage buyers to have similar attendance rates right around 30%.
And when we let early stage folks go ahead and sign up organically, their attendance rates shoot up from 15% to 26%, which is great!
It’s all about investment
Give people what they want, when they want it
Put more focus on early stage
NO RE-ENGAGEMENT STRATEGY
Don’t you just love email design challenges? There’s nothing better than having an element render differently across all devices, right?
So we use HTML buttons in our email templates, which are buttons made out of code. Here are two screen shots of the same email from different email clients. As you can see, the button likes to take many shapes and sizes. On the left, the text fills nearly the entire button, appears bolder, the space between the first paragraph and the button is squished, and the overall button size is smaller.
On the right, it at least appears more aligned within the button and the copy itself, however the text looks a little small floating inside of a big button.
I’m a bit of a perfectionist and I love to problem solve, so this finicky button became my main focus for a few weeks.
The question I was trying to answer was “Does CTA button treatment affect click to open rates?” My goal was to create a button that looked a little cleaner but also had more robust code to keep it aligned in every email.
So, which button would you rather click?
The control has squared corners, all uppercase text, and looks a little antiquated if you ask me. Plus, all caps just feels like yelling to me and I wanted to fix that.
The test has rounded softer corners, proper case italicized font, and a few code additions to stiffen up it’s rendering to be more consistent.
Just a few minor changes really, and my real hope was that the click to open wouldn’t go down at all and I could just update the buttons to make them look prettier. Because we all know that as marketers, we don’t always feel good about our work unless it looks pretty.
So this button has more robust HTML to make sure it stays in place on even the worse of email clients.
Button Attributes:
Table expands to 100%, not a fixed width
5 pixels of padding on top of the button
10 pixels of padding within the button to keep text centered
All in all, these additions will help to keep the button consistent across email clients, browsers, and mobile devices. Keep in mind too that although these may seem like small updates, emails that don’t render correctly don’t get clicked. As an email marketer, I am super critical on email templates and if something appears broken in my inbox, I move on to other more important emails.
Here’s a screen shot of a rendering with the new button. What do you think? Prettier? Softer on the eye? I think so. Now let’s dive into the testing details.
Results by significance:
New button has a 4% higher click to open (statistically insignificant)
New button has a 25% lower unsubscribe rate than the control (100% statistical significance)
The rounded button with italicized text and proper spacing reduces our unsubscribe rate by 25%!
Sometimes you test for one thing and solve for another
Use well-built HTML buttons (you can find them online)
Alright, that’s all I have for you. Before I answer a few questions, I’d like to remind you that there is a brief survey after this webinar. Please take 30 seconds to complete it to let me know how we can make these better for you in the future.
Now on to the questions!
What do you think makes a bigger impact: subject line testing or body copy testing?
How do I make an HTML button?
Have you run tests on best day of week to send an email?