Test for Success: A Guide to A/B Testing on Emails

Jessica

jessica

Jessica

Jessica -

We’re marketers, not mind readers. The purpose of A/B testing is to help you determine which version of your email or website converts the best. It’s an inexpensive way to generate more leads, without having to send out more communications. Even minor changes can produce a huge impact - but more on that later.

Jessica
It’s harder than ever to get your message heard through all the noise - so you need to be as effective as you can.

On any given day, the average customer will be exposed to over 3,000 media messages, they will pay attention to 52 of those messages and will positively remember 4. This is why we as marketers need to become even better at capturing the attention of our audience and hopefully become one of those 4 messages that they remember.

Jessica

Note that it is entirely possible, and natural to have more than one company objective. An example is when working

Once you have identified your conversion metrics that you want to follow, you can identify how different pages on your site affect that objective.

If you’re looking to increase certain purchase numbers, like Electronic Arts, they smartly chose to A/B test their product page. Another example is if you are looking to increase number of trials started within a certain product.

Look for pages that have high traffic, and high impact. High traffic pages, with a lot of visitors, will ensure that you reach accurate and effective results from your test as quickly as possible. Pages that have high impact on your business are pages that are an integral part of your company’s goals.

Some examples of high traffic/high impact pages are your home page, a product page, or a form page, like Contact Us or Request a demo.

You could also look at the search results page, article page, or pricing page.

Now that we have identified a crucial step in your conversion goal’s flow, we now can choose what component of the site or email that we want to test.

With so many components to a single email or webpage that you can test, are there some components that are better to start with than others?

When you are first starting out with A/B testing, a couple easy guidelines to get going is to reduce noise on your site or email, while increasing focus to your goal.

This is exactly what Electronic Arts did in removing their hero image. By increasing focus to the true call to action (purchasing pre-orders), moving up the buy button further up the page, as well as reducing noise on their site, they were able to successfully increase purchase conversions!

If you are getting started with Email A/B testing, a great first test would be to test the subject line, using the same principles: reduce noise and increase focus. A great example of a test we here at Optimizely did was test out our subject line by explicitly highlighting what type of content we were providing in our promotional email. In doing so, we increased the focus for our users, reduced marketing jargon noise in our subject header, and as a result increased click-through rates.

While there are many components to test within your page/email, start with something that will increase focus to your end goal. We will get into a lot more examples of types of tests you can run, but here are some good components to get started with.

Hypothesis-driven testing means that you start by identifying what experience you want to change and how it will affect your business results -- not just changing things and hoping for the best.

If you’ve done the first three steps, creating a hypothesis is not difficult. And I’m not telling you to state your hypothesis because like the Secret, if you say it, and mean it, it will be true. Stating a hypothesis encourages you to define exactly what you are testing, and with what goals, so that you can stay on track, and successfully report on that hypothesis when the results come in.

Whenever I launch a new program, the first piece of the puzzle I always tackle is: what are my reports going to look like? By identifying exactly how I want my reporting to look, I have a better and more strategic idea of how I want to implement this new program. Stating your hypothesis identifies the particular component that you are testing, as well as the exact conversion metric you want to improve, as well as the scope. That way, when you are reporting on this A/B test down the road, you know exactly what to look for, and whether the A/B test was success, leaving no room for error.

On top of that, stating a hypothesis reinforces internal team alignment, by putting the cards on the table, and leaving no doubt exactly what is being tested and what goals are being tracked.

Hypothesis-driven testing means that you start by identifying what experience you want to change and how it will affect your business results -- not just changing things and hoping for the best.

If you’ve done the first three steps, creating a hypothesis is not difficult. And I’m not telling you to state your hypothesis because like the Secret, if you say it, and mean it, it will be true. Stating a hypothesis encourages you to define exactly what you are testing, and with what goals, so that you can stay on track, and successfully report on that hypothesis when the results come in.

Whenever I launch a new program, the first piece of the puzzle I always tackle is: what are my reports going to look like? By identifying exactly how I want my reporting to look, I have a better and more strategic idea of how I want to implement this new program. Stating your hypothesis identifies the particular component that you are testing, as well as the exact conversion metric you want to improve, as well as the scope. That way, when you are reporting on this A/B test down the road, you know exactly what to look for, and whether the A/B test was success, leaving no room for error.

On top of that, stating a hypothesis reinforces internal team alignment, by putting the cards on the table, and leaving no doubt exactly what is being tested and what goals are being tracked.

Average order value will increase
Lead forms submitted will increase
Number of purchases will increase

It takes time and statistical significance to declare a winner for an experiment
If your variation didn’t win, that’s still valuable data
If you’re not getting significant results, try making bigger changes or using a page that gets more traffic

Jessica

Most email clients that allow you to A/B test will aggregate the results like this screenshot here. This particular test was on the From Name like the test I mentioned before. The first email was sent from Marketo Premium Content while the bottom one used tokens that populated with the lead owners name. In this case, the metric we were basing success off of was open rate and as you can see here, the email sent from the lead owner had a 17.2% open rate, versus premium content which had a 15.4% open rate

If you’re not comfortable with finding significance on your own in Excel there are free calculators online that you can use. This particular one is on ZettaSphere.com and all you have to do is plug in the values that you were tracking and your sample size and it will tell you how significant your test was

Or if your test is not significant, it will tell you to modify your test or your sample size in order to reach a more significant conclusion.

~8-10 minutes total

Subject Line
[brackets]
From line
Button Copy
Button Placement

When it comes to A/B tests, there are so many different variables to manipulate. You can think about font, text size, color etc. We’ve tested using speaker headshots in webinar invites vs. not and found positive results for including them which is why you all probably got in invite that included my and Allison’s faces in it.

Jessica

There are lots of things you can test on emails and landing pages. After this webinar, we’ll be sending a link to the recording as well as this joint cheat sheet from Optimizely and Marketo on the 30 things you can start A/B testing to increase lead generation.

Now we’ll move on to how to declare a winner.

Allison

Cara