9 Things You Might Not Know About A/B Testing

Image for post
Image for post

A/B testing is a hot topic in marketing now that data is such an integral part of our marketing efforts.

This method of testing is used to optimize and streamline the user experience on an email, landing page, or web page.

You may understand the concept of A/B testing, you may have even run a few A/B tests on your own. But do you know every one of these surprising A/B testing tips?

One of the first things you define in A/B testing is your goals. The final goal, of course, is the conversion. This means the purchase typically. Optimize the very end of your funnel first.

Why? These optimizations make the biggest impact. They directly affect your goals and will ensure that your leads aren’t finding friction at the very end of your funnel.

You must have goals. That means figuring out what you want a user to do when they open an email, read a post, or land on your website.

Before you start trying to run A/B tests willy-nilly, you need to know what goals you’re interested in optimizing and which elements of your test page affect those elements.

It might be tempting to attempt to optimize for every page, but you need to focus on the highest impact pages, especially when you first get started.

What does that mean? It means:

  • Pages near the bottom of the funnel
  • Contact pages
  • Landing pages for email campaigns
  • Your homepage

To have a truly successful A/B test, you need to run it like the actual scientific experiment. You can’t just wing it and try to randomly test elements.

You can work off the following list of steps:

  1. Find problem areas in your funnel — these are areas you want to optimize.
  2. Find issues using user behavior — Figure out what is stopping users from converting.
  3. Create a hypothesis — this should be a short sentence that describes a change you think will increase conversions. For example, “Changing the CTA button to red will increase conversions.”
  4. Test your hypothesis — design a new page with a red button and split your traffic so half goes to the control and half to the test page. Halt your test when it’s clear you have a winner. We’ll go into that more later.
  5. Analyze the results and draw conclusions — from that A/B test, what did you learn. Did the change you make increase conversions in a statistically significant way? If the test is inconclusive, you might have to restart the test, starting at your hypothesis.

While it might be tempting to change a bunch of things at once, then you don’t know what your users prefer. If you change the size of the button and the color of the button and you see increased conversion, which of those variables caused it.

There’s no way to tell in a simple, single variable A/B test. Just make one change at a time. You’ll learn what your users prefer, which will optimize both the page you’re working on and other pages you may work on in the future.

If you do a test on one variable, then add another, be sure to start a new test. You don’t want to just add a new variable and call it a day.

It will make it almost impossible for you to determine when to stop the test. On top of that, you won’t be able to determine which of the variables worked and which didn’t.

Keep it simple and test one thing at a time.

How long is long enough for an A/B test?

There are four factors to running a “long enough” A/B test.

  1. You reach a sufficient sample size. This will have to be calculated before the test occurs. There’s a tool for this.
  2. The test has run for multiple sales cycles (think 2–4 weeks)
  3. You reach statistical significance. That’s a formula that determines whether a result is from chance or from fact. The higher the percentage, the more likely the result is the real deal. You can download an extension that will allow you to calculate this on Google Analytics, or you can work with the formula in Excel and find it yourself.

Cloaking is the practice of showing one set of content to visitors and another to Googlebot.

This can have devastating effects on your website; it can even get your site demoted or removed from Google search results.

To avoid this, don’t serve the test based on user-agent.

Use 302s Instead of 301s

If you’re redirecting traffic from the original URL to the test URL, use a 302 redirect. It’s temporary.

Don’t make the mistake of using a 301 redirect because that’s the permanent one. Your test is temporary, so you don’t want to permanently change where the original URL goes.

Written by

HIPB2B is a demand gen solution provider, utilizing content, email, and marketing automation to drive outcomes for marketing and technology clients

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store