If you don’t know exactly what A/B testing is, check out this short read before you read this.
So now you know what A/B testing is, and why it is crucial for your product (if not, learn it in 3 minutes by reading this post), and the next natural step is to learn how to integrate it into your product management processes. A/B testing can be implemented in several ways, and I am sure you will hear different tips, from different sources. I will introduce you to the different steps I would recommend you to follow, to perform reliable, efficient, and useful A/B tests.
Step 1: Think deeply about your product area
When you’re trying to understand what you want to A/B test, think of the product area, that you’re working on. Have a holistic view of it and ask yourself, which parts of it you believe will work better. And why? You don’t have to but we advise you to get yourself some data to help you make decisions and to understand how your product is being used. You can do this by using heatmaps (crazyegg, Hotjar), other analytics tools (Google Analytics…), or running usability tests with a sample of your customers and users.
Step 2: Analyze your Information
Now that you have gathered data about your product areas, decide where you want to start. Observe areas that you think could improve some kind of conversion rate, be that a call to action, color, images, or headlines. You can simply change a button or make major changes, as a complete sign-in page. It’s your call! The most important thing for you is to understand the “why” of making those changes and define your hypotheses.
Step 3: Write down your hypothesis
A hypothesis is a prediction you’ll define before start running your experiment. It protects you from your own bias. Your hypothesis is a description of why are you going to change something, what are you changing (your variant B), which customers or users are you aiming to benefit from it, and how will it be doing it. Then describe your expected outcome and how it will align with your business goals.
Running an A/B test without a hypothesis is like throwing darts in the dark, not knowing what you’re aiming for.
Step 4: Define the duration of your test and be ready to run it
In this phase, you want to learn how long you need to test so that you can rely on the results to make conclusions and make decisions. The whole goal is to understand how many users need to use your product in order to achieve statistical significance. Once you understand this, it will be easier to forecast the length of the test, as well as its cost.
To simplify this there are many calculators online, and I would recommend the A/B Test Sample Size Calculator from Optimizely.
If you expect a high conversion rate change, you’ll not need as many users in order to get results that are statistically significant. On the other hand, if you expect a low conversion rate change, you will need to test it with a larger sample of users. After you calculate this, it’s easier for you to understand how long and how much will it cost to run your experiment.
If you’re up to follow up with the test, release the new variant and wait to see the results.
Last step: Conclusions
Now is time to analyze your results. Compare the different variants in your experiment, and take your own conclusions and next steps!