Join the conversation on LinkedIn or @MarketingB2B on Twitter with #DearB2B. You can also email [email protected] to get help with your toughest B2B marketing challenges.
Q: Dear B2B, what are the biggest mistakes marketers make during A/B testing and how can these be avoided?
A: A/B testing is a popular and effective way for marketers to accelerate conversion rate optimisation and harness the power of creative to improve the usability of their website and drive traffic. However, contrary to what many may believe, optimisation’s not as simple as adopting efficient A/B software. Even the most experienced marketers make errors without even realising.
Here are three major pitfalls to avoid:
1. If you’re early stage, go big. If you’re late stage, control and optimise
One of the biggest mistakes that early-stage products or companies make is being too scientific – i.e. holding all variables consistent except for a single variable. There are a variety of variables that can be tested, such as sender name, subject line, and text versus HTML. When volume is low and you’re searching for directional wins, make sure you test strategies or templates, not modules or copy. But, when volume is high, do the opposite and focus on copy or modules. When testing at scale, even 1% improvements are statistically significant and can equate to bigger improvements down the line.
2. Avoid cutting tests short
Ending an A/B test early or ending it as soon as it has achieved statistical significance are other ways in which marketers commonly block themselves from seeing positive results. Many will end a test early and then not account for a representative sample.
For example, if you start a test on Monday and end it (with significance) on Friday, then you didn’t account for weekend buyers and buying habits, which might differ from any other given day of the week. By determining the minimum sample size necessary for the study in advance and increasing the time span of your test to last through multiple business cycles, you improve your chances of capturing the effects of plenty of variables.
3. Be wary of adopting results from other conversion rate optimisation case studies
With the vast number of B2B marketing case studies online, it’s sometimes tempting to simply copy the results of another company’s A/B test – whether it be around the timing of an email, HTML versus text email templates, subject lines or micro-copy changes. But every company has a different target market and audience. Plus, case studies often fail to include the entirety of the data found through the course of the study, or may be filled with inaccuracies.
If possible, perform your own tests to avoid using insufficient data. By doing this, you’ll have full control over the data and analysis with contextual information that’s tailored to your company’s unique goals and objectives.
