10.02.2021

Marketing & Growth

Should you split test or AB test marketing & sales emails?

12 min read

Verity

Share on

A/B testing - or split testing - email marketing is one of the few practices marketers employ to improve the performance of their email campaigns. How to split test your emails to improve the response rates of your email marketing strategy.

 

Understanding and utilising concepts like multivariate testing and conversion rates can significantly uplift your campaign's success.

This article explores the nuances of email optimisation through A/B and split testing, emphasising statistical significance and the role of testing tools in refining your marketing strategies.

Understanding the basics: A/B and multivariate testing

Before diving into the intricacies of email optimisation, it's crucial to differentiate between A/B testing (also known as split testing) and multivariate testing.

A/B split testing involves comparing two versions of an email - the original version and a variant, to gauge which one performs better in terms of conversion rates.

Multivariate testing, on the other hand, takes this a notch higher by examining how multiple variables interact and affect user responses.

There is a delicate balance between single-element tests and complex multivariate tests. For instance, changing a single element like button colour might offer clear insights into what works better.

However, when combining multiple variables such as design elements, email subject, and even the layout of action buttons, the analysis becomes more intricate but potentially more revealing, especially when assessing statistical significance.

The methodology also extends to understanding your audience. Through split testing tools, marketers can segment their target audience, tailoring messages to different groups based on their preferences and behaviours.

This segmentation not only enhances the relevance of emails but also helps in fine-tuning the overall marketing efforts.

What is statistical significance?

Statistical significance is a term used in statistical hypothesis testing to help determine if a result is not likely to occur randomly, but instead is likely due to a specific cause. 

In simpler terms, it's a measure of whether the results you are seeing in your data are meaningful or could have happened by chance.

When conducting a test, such as an A/B split testing in marketing, researchers set a significance level (commonly 0.05, or 5%). 

This level represents the probability of observing a result as extreme as, or more extreme than, the observed data, assuming that the null hypothesis (the hypothesis that there is no effect or difference) is true.

If the p-value (the probability of obtaining the observed results, or more extreme, under the null hypothesis) is less than the set significance level, the result is considered statistically significant. 

This means that it is unlikely the observed effect or difference is due to chance, and there is evidence to suggest that a real effect or difference exists.

For example, in the context of email marketing, if you conduct an A/B split test to see which email subject line yields a higher open rate, and the result is statistically significant, it suggests that the difference in open rates between the two versions is likely due to the change in subject lines and not just random chance.

The role of conversion rates in email optimisation

At the heart of email testing lies the goal of conversion rate optimisation.

Whether your focus is on the email subject line, the design of action buttons, or even the button colour, each element plays a critical role in influencing conversion rates.

By methodically testing variations of these elements, you can understand what resonates best with your target audience, ultimately leading to improved marketing campaign effectiveness.

Leveraging testing tools for insightful decisions

The market offers a plethora of testing tools, including specialised split testing tools and comprehensive platforms like Google Analytics or HubSpot.

These tools provide invaluable insights into user behaviour, helping you tailor your emails for maximum impact.

They enable you to track user interactions with different elements of your email, be it the control version or a newly introduced variant, and measure these against your conversion goals.

Analysing results: from data to decisions

The crux of successful email testing lies in analysing the results. Tools like Google Analytics and HubSpot play a pivotal role here.

They help in tracking various success metrics, from basic open rates to more advanced metrics like user engagement levels.

By correlating these metrics with specific changes in the email's design or content, marketers can make data-driven decisions to optimise their campaigns.

Best practices and advanced concepts in email testing

As we venture further into the realm of email optimisation, certain best practices emerge as vital.

Firstly, maintaining a representative sample size in each test ensures that the results are reliable and can be scaled to a larger audience. This is particularly important when dealing with complex multivariate tests, where multiple variables are at play.

Another advanced concept is understanding the sales funnel and how each stage might require different email strategies.

For instance, emails aimed at the top of the funnel should focus more on user engagement and brand awareness, while those at the bottom might need to be optimised for direct conversions, like form completions or ancillary purchases.

The impact of social proof and user experience

In today's digital landscape, incorporating elements of social proof, like customer reviews or social sharing buttons, can greatly enhance the effectiveness of your emails. These elements not only increase trust but also encourage user interaction, contributing to a more engaging customer experience.

Furthermore, optimising emails for different devices, particularly for mobile users, is crucial. This includes considerations for visual elements, button size, and overall layout, ensuring that the user experience is seamless across all devices.

How to conduct a split test on HubSpot?

Hubspot makes it super easy to split-test your regular emails (emails that are sent as a one-off).

To set up your A/B split test:

  1. Prepare for your A/B split test:

    • Determine the goal: Before setting up your test, decide what you want to achieve with your email. This could be increasing engagement, responses, clicks to the website, or driving sales.
    • Select elements to test: Based on your goal, choose elements to experiment with, such as the email subject line, imagery, content, or calls to action.
  2. Create your A/B split test emails in HubSpot:

    • Navigate to Marketing > Email in your HubSpot account.
    • Click "Create email" to make version A of your A/B split test.
    • In the top left of the editor, click to create an A/B test. Enter a name for the variation B email and click OK.
    • Consider varying aspects such as offers (e.g., eBook vs. whitepaper), copy formatting and style, email sender (personal vs. generic address), imagery, subject line length or personalisation, and the entire email layout for comprehensive testing.
  3. Configure test settings:

    • Set A/B distribution using a slider to decide the percentage of contacts that will receive each version. You can opt for a 50/50 split or other distributions.
    • Choose a winning metric (open rate, click rate, or click-through rate) to determine the more successful variation.
    • Set test duration to gather data before deciding on the winner. If results are inconclusive, HubSpot defaults to sending Variation A to the remaining recipients.
  4. Finalise and send the test

    • After configuring the test settings and reviewing your email, click "Save changes."
    • Review your email settings and configure your recipients, then click "Review and send" to initiate the test.
    • Note that you can run a full-split (50/50) A/B test with any sample size. Other sample splits require a list of at least 1,000 recipients, excluding contacts that previously hard-bounced or unsubscribed.


A/B Testing Screenshot

Examples of elements you can test:

Subject line

When you split test email campaigns, the subject line is always a good place to start. You can experiment with capital letters, emojis, questions, and shorter or longer sentences. This helps to establish how your email open rate is affected.

Imagery

Imagery is a subconscious area and the images that you choose have an impact on your readers’ responses. 

Testing different types of images is always a good way to see what content your readers are attracted to and engage with the most. 

Try testing differences such as image size, people vs non-people, bold or pale colour palettes and humour.

Content

Your brand messaging and the information that you share all make an impression on your audience. 

Without wandering too far from your company voice, you can experiment with altering your choice of language, colloquialisms and tone of voice to see if it increases your click-through rates.

Call-to-action

Once a subscriber has opened your email, it’s essential to have an effective call-to-action (CTA) so they take the intended next step. 

Often, people limit CTA usage to simply changing the button or text colour, but there’s a lot more than you can test to improve your click-through rates.

  1. Location: The location of a CTA can make a big difference in click-through rates. If people don't scroll down to the end of your email, they might miss it. So, why not try it out in different places to see what kind of change that makes?
  2. Images: Experiment using image CTA’s instead of just a button with text. Images are likely to attract more people and may encourage them to click through.
  3. Wording: Experiment with changing up the text on your CTAs.BUY NOW may work for some communication, but for prospects who aren’t at that stage of the buyer’s journey, you may have more luck with BROWSE MORE. Your audience can tell you what works best for them, and the stats will show this after you send the email.

How Huble can help you optimise your email campaigns

While the strategies and insights outlined in this article provide a robust foundation for enhancing your email marketing campaigns, the journey towards optimisation is continuous and dynamic.

This is where Huble steps in. As a seasoned marketing consultancy, we specialise in helping businesses like yours navigate the complexities of email campaign optimisation.

Leveraging tactics like split testing, multivariate testing, and a deep understanding of user engagement and conversion metrics, our team is adept at crafting email strategies that resonate with your audience and drive results.

If you're ready to elevate your email campaigns and harness the full potential of targeted, data-driven marketing, we invite you to speak with our team.

Let's explore how we can help you achieve your marketing goals and transform your email initiatives into powerful tools for business growth.

Book a free SEO audit

Your website is your most valuable sales and marketing tool, so it needs to be as optimised as it can be. Uncover opportunities to boost your rankings with a free SEO audit.

Heading Text

Sales & CRM

37 min read

HubSpot security and compliance: best practices and automated tools

Explore how to leverage HubSpot’s security tools and resources to mitigate risks and ensure Security and Compliance in HubSpot.

Read more

Culture

3 min read

Huble named #1 in HubSpot Consultancy Proficiency and Marketing Strategy Agency on G2

We are proud to have been recognised as the leading solutions provider in the Best HubSpot Consulting Services category of the G2 winter reports. 

Read more

Marketing & Growth

22 min read

Everything you need to know about HubSpot Content Hub

Learn all about HubSpot's Content Hub, including the key benefits and pricing, what's changing and how Marketers can utilise the platform.

Read more

Sales & CRM

25 min read

HubSpot Data Management: 6 strategies for success

This article dives into the central role of data management in maximising CRM potential and best practices for data management in HubSpot.

Read more

Marketing & Growth

50 min read

112 Account-based marketing statistics you need to know in 2024

In this article, we look at 113 account-based marketing statistics that showcase how this strategy has transformed modern marketing.

Read more

Sales & CRM

10 min read

3 sales team structures that set you up for success

Is your current sales team structure helping you do your best work and hit those goals?

Read more

Content Strategy

9 min read

Mastering B2B brand messaging for effective demand generation

This article dives into the essential nature of B2B brand messaging and provides actionable insights on crafting messages that engage with B2B buyers.

Read more

Sales & CRM

12 min read

Enhancing CRM security in HubSpot: beyond traditional measures

Explore both the basic security measures that lay the groundwork for CRM security and more advanced forms of CRM security, in particular in HubSpot.

Read more

Marketing & Growth

10 min read

Is it worth it: What is the ROI of HubSpot Marketing Hub?

Learn more about the ROI potential of HubSpot’s Marketing Hub and how the platform can lead to higher productivity, cost efficiency and customer loyalty.

Read more

SEO & Paid Media

8 min read

Third-party cookies are dead: what HubSpot users should know {Mar 2024}

In this article, we’ll delve into the demise of third-party cookies and explore Google's recent actions to phase out third-party cookies.

Read more
Button Text