A/B Testing Mastery
A/B testing puts effectiveness as its top priority by experimenting to see if something has the desired impact or not. For example, a company can use it as a tool of experimentation when deploying something new on their website. It can also be used in research too. The A/B Testing Mastery course is taught by Ton Wesseling, a well-known dutch expert in the world of digital optimization and conversion.
This course is a real lesson on how to master the A/B testing process in general, going from the more methodological aspects to the more technical and technological aspects. A course for which having some basic knowledge of statistics is not a bad thing.
An approach to A/B testing
A/B testing is nothing new. It is something that historically has been seen in fields such as medicine, but it is in 1995 when it began to be applied to the world of digital products and it is in 2013 when its use spreads due to the emergence of simpler tools that do not require the use of programmers to carry out A/B tests.
This is how this methodology becomes popular and begins to be used more and more by all companies, and this tool, worked together with an Agile methodology has the ability to produce better results than when we use a Waterfall style approach.
A/B testing allows us to be more accurate when developing digital products through a paradigm shift when working on the improvement process. Now we do not have to base our decisions on opinions but on data that come from the reality of how our users interact with our digital products. And that is why the more decisions we make based on data produced by experiments, the higher the quality of the evidence and the lower the risk of using our own biases and those of our teams and stakeholders.
When to use A/B testing?
It is advised not to run experiments with less than a thousand users, since that would represent a low statistical precision in our experiment. And operating with figures lower than these numbers would not give us completely conclusive results.
And for this, we have the ROAR model. A framework that relates conversions per month over time. And it does this in four phases. Risk, if you have less than a thousand conversions per month, Optimization, if you are between 10,000–10,000 conversions, Automation, when you have more than 10,000, and you have already validated your experiments and Re-think.
He also introduces the term Statistical power, which basically is the likelihood that an experiment will detect an effect when there is an effect there to be detected. A power that depends on the sample size, effect size and significance level.
It also clarifies the importance of being accurate at the moment in which we launch our experiments, one of them may be when we have a lot of traffic and a low level of conversions for example. We have a large sample from which to extract data and a well-defined problem to focus on. Since here we could determine a series of elements to change, such as the value proposition, design elements, or the hierarchy of information on the landing page to analyze user behaviour.
And with A/B testing we have the ability to measure at different times and in different places all those variants that we apply to our products to improve them. And to start them we only need to determine if we have the necessary traffic and what KPIs we will measure.
What to measure?
And this is when we talk about defining our “north star metric,” “One metric that matters,” or “OEC-Overall evaluation Criterion,” as he calls it because it’s really important to have a Goal metric for the whole company. A shorten metric to align a long term value for the company goal. Or could be a combination of several metrics.
These could be Clicks at the less important KPI, Behaviour is the second layer is an interesting metric, Transactions (leads B2B, transactions in B2C) or also called the golden metric, Revenue per user or the Potential Lifetime Value, the most important metric. Once we have defined what to measure, we can enter the real process of experimentation.
How to use A/B testing
Once the objective is clear, we need to do a desk research to guide our experiment and understand the main user insights to create a solid hypothesis.
The hypothesis is the basis of the scientific method. Ton invites us to develop A/B testing through the scientific method applied to the experimentation and optimization of digital products. A methodology that consists of the following steps:
FIND >ANALYZE> CREATE >TEST> ANALYZE >COMBINE >TELL
And to start with the first phase of research we present the “conversion canvas” with the following points to take into account to develop our desk research that has 6 points. 6 points to take care of at the moment to investigate our user insights.
>VALUE. What company values are important and relevant? What focus delivers the most business impact?
>VERSUS. Who are your competitors and market best practices?
>VIEW. What insight can be found from data analytics and data behaviour?
>VOICE. What insights can we take from a voice from user data?
>VERIFIED. What scientific research or data are available? Which insights are validated in previous studies that we have?
This research will allow us to dispose of our hypotheses. Because without analyzed reason to start don’t have any sense to start at all. And if you wonder how to create a hypothesis you only need to establish a correlation between these three points:
1-Define Problem
2-Proposed solution
3-Predicted outcome
And once we have created our hypotheses we only have to prioritize them with the method we prefer, either PIPE (potential, impact, power, ease) or ICE (Impact, Confidence, Effort).
Now that we have the hypotheses, we will prioritize them and launch our A/B tests on our digital platforms. To do this, we will prepare them on our platforms, monitor them during the process, and observe the results obtained in order to analyze them. A series of more technical steps that I invite you to review in CXL Institute’s A/B testing Mastery course. Really, I´ll recommend you.
What to do with the results
Once these results have been obtained, Ton tells us that it is very important to know how to present them to our organization or company so that they can make business decisions. And one of the best ways to do this is through business cases. That is why Ton also dedicates a class to this step.
Finally, he also gives us an introduction to how to manage a CRO team within a company and at what moments we need to scale our team to be able to scale our experiments as well. And when experiments grow, they also need to grow the staff needed to carry them out.
Full disclosure: I’m taking the CXL Growth Marketing Minidegree as part of a scholarship, on the basis that I write one review post per week, over twelve weeks. If you’re interested in reading a really in-depth review of the Minidegree, keep an eye on my Medium page over the next 3 months. 🙂
