Why A/B Testing is so Important in E-Commerce

1. What is an A/B test (in e-commerce)?
2. How does A/B testing in e-commerce work?
3. What exactly can you check with A/B testing in the online shop?
4. What are the Advantages of A/B Testing in E-commerce? Why is it Important?
5. How Can the Success of A/B Testing in the Online Shop be Measured?
6. What Tools are Suitable for A/B Testing in E-commerce?
7. Conclusion on A/B Testing in E-commerce

A/B testing is a proven method for making long-term, data-driven improvements to your online store. Especially in e-commerce, A/B testing is absolutely necessary and it's no coincidence that companies like Amazon have integrated this topic into the core of their growth strategies. Our guest author Marven Hennecker shows you in this article what you need to consider when it comes to A/B testing.

What is an A/B test (in e-commerce)?

Basically, an A/B test in e-commerce is a pure random experiment in which visitors to an online store are shown at least two different versions of the online store. The selection of which version is ultimately shown to which visitors is based on chance. The goal of an A/B test in e-commerce is usually to verify assumptions and make decisions based on data-based results. Such an A/B test is always carried out in a suitable testing tool, more about this in the further course of the article. Additional general information can be found in the article on the A/B Testing Basics..

How does A/B testing in e-commerce work?

Starting off is always tough. This is true when you want to start your first A/B tests in your online store. Basically, the rule here too is that many roads lead to Rome. But what is needed for this journey?

In this section, you will get a step-by-step procedure that will help you start A/B testing and take the first steps on your own. Please be aware that this approach represents our best practice model and may not apply to every online store.

Finding Test Ideas

First and foremost, you need test ideas. Because every A/B test always starts with a good test idea. It is important to determine that a test idea does not have to have a complex structure and you can let your imagination run wild. However, you should always make sure that the test idea has a starting point for possible improvement. Alternatively, you can objectively question elements of your online store. Because most importantly: Personal opinions and feelings should be set aside at all stages of A/B testing. In the picture below, you can quite easily understand what we mean by this.

Good test ideas are usually generated differently, specifically by analyzing your qualitative and quantitative data systematically. Your online store is constantly producing data, which you can find in your shop system or your analytics program. Furthermore, a review tool, like REVIEWS.io or Trusted Shops can provide additional starting points. You can also use a tool like Hotjar to analyze user behavior and generate test ideas. You should always try to generate as many test ideas as possible. Therefore, there is no specific number that you should aim to achieve.

To make it easier for you to generate test ideas, we have created a small graphic where you can find 15 test ideas for a product detail page. How many can you find?

15 Test Ideas

Formulate and prioritize hypotheses

It is important that your test ideas convey structure and meaningfulness. Especially with a large number of generated test ideas, this point is extremely helpful in order to provide your test ideas with comparability in the long run. Therefore, in the next step, we will turn all test ideas into concrete and precise hypotheses.

But first, we should clarify what a hypothesis is. A hypothesis is a supposition that is assumed to be true for certain purposes until it is confirmed or refuted. It is therefore a justified assumption about a possible outcome.

Creating these hypotheses does not necessarily have to follow a specific procedure, but generally it is recommended to follow a framework. We recommend the If-Then-Because framework.

• IF = specific change in the online store
• THEN = Expected result (KPI)
• BECAUSE = Detailed and substantiated reasoning

We are happy to demonstrate this framework using a very practical example:

You want to start an A/B test on a product detail page because you could detect an anomaly in the bounce rate of this page. Therefore, you want to test a new element focusing on social proof and formulate the following hypothesis:

• IF we display the average overall product rating near the price on the product pages
• THEN the Conversion Rate rises
• BECAUSE customer trust is increased by the effect of social proof and the pain of price decreases

After we have now understood how test ideas become concrete hypotheses, it could quickly be assumed that we can hop onto the next step. Unfortunately, that's not the case, we still have to refine the hypotheses we have created.

Let's say you've formulated a total of 20 hypotheses. All these hypotheses differ in terms of the likelihood of achieving the expected result. Ultimately, only the most promising hypotheses should be tested.

That's why it's important for you to make your hypotheses comparable and prioritize them based on this comparability. Again, a framework helps you out here, but you have to align it with your own dimensions. We use a system of eleven criteria, for example, where we record, among other things, whether the test takes place "Above the Fold", i.e. in the area that is visible to customers without scrolling. If so, the test has a higher priority because it is seen by a large number of visitors.

Another criterion for us is, for example, whether the idea was generated by a quantitative data analysis. If so, the priority is also higher here than without a data basis behind it. Ultimately, it's about working out the most promising test ideas and hypotheses for you with the prioritization. The eleven criteria of our framework help us achieve exactly that.

For starters, however, you can also start with the so-called "PIE framework":

• P stands for Potential, i.e. the success probability of the test.
• I stands for Importance and estimates how relevant the test should be at the specific point of the funnel.
• E for Ease and tries to express how quickly such a test can be launched and how much effort is associated with it.

Each of the three letters usually gets an estimated value of 0–10 assigned and at the end the average is calculated. 

Suppose you award P = 7, I = 9, E = 2, we come up with an average of 6 scoring points. Each hypothesis should be evaluated in this way and the ones with the highest scoring can take the next step first.

Design and program the test.

We have been on a very good path so far and are getting closer to your first A/B test, well done. After we've identified the most promising ones, it's time to bring the hypotheses to life and then program them.

In the first step, we should start turning the textual form of the hypothesis into a visual one. Your hypothesis only sets the framework conditions for your A/B test, but it is not a 1:1 guide for implementing it. This means that you can also let your creativity run wild here. However, always keep your hypothesis in mind during each design progress and keep asking yourself whether this change or this element can be reconciled with it. Technically, you should always design your designs via a tool like Figma or Adobe Photoshop. Make sure you spend a lot of time on your designs and make an effort here, as your designs are always the blueprint for programming at the end. Even with your designs, you should always create at least two different proposals, objectively evaluate them, and finally decide on one.

Compare the designs

Then the A/B test needs to be programmed in a suitable A/B tool according to the design instructions. This should be definitely done by a person with relevant expertise. As soon as this step is finished, your first A/B test can go live. Congratulations! 

PS: No one likes faulty presentations when buying online. Therefore, always check the usability and quality of the programming before the A/B test goes live.

Evaluate the test and use results for new approaches.

Once the test is live, things get exciting. It's important to keep a close eye on the goals you're measuring. More on this in a later section.

First of all, you calculate a so-called Minimal-Detectable-Effect for the test runtime and then know exactly how long the test needs to be online to achieve a statistically significant result. Statistics are very important in A/B testing, even though the topic is often not so popular. Therefore, it is essential to deal with it during A/B testing and to delve deeper into the topic.

Because only if the calculations here are done correctly, the test can later actually be judged as a winner or loser. After the calculated runtime, the test is then turned off and the significance level is determined.

There can be the following three situations:

1. Your variant performs significantly better than the original

Congratulations. You have completed your first successful test and will thus generate a monetary added value. The variant should therefore be permanently visible in the shop instead of the original. Keep this test in mind for the future, as you can probably use the hypothesis in a similar form in the future and apply it elsewhere. 

2. Your variant performs significantly worse than the original

Here too, you have a significant result and can learn from it. Your hypothesis did not work. Now it's important to consider what caused this and how you can improve the next tests with a conclusion.

The variant should definitely not continue to run in the shop instead of the original. 

3. Your variant shows no significant change compared to the original

This is usually the worst-case scenario because you don't know what caused the variant to perform neither better nor worse. But please avoid seeing the test as positive or negative just because there's a tendency in one of the two directions.

What exactly can you check with A/B testing in the online shop?

In theory, it is possible to check almost every single component of your online shop in an A/B test. But generally, you should only A/B test things that you can reasonably expect to be able to improve.

A/B Testing Examples

Various pages (PDP, category page & Co.)

Especially on start, category and product detail pages, you can A/B test a lot of different things. To provide concrete added value, we want to give you the following ideas.

Homepage:

• Check your top bar for significance, content, and process
• Check whether the page starts with social proof or with certain products
• Check the display and access to the menu (e.g. a pictogram menu)

Category page:

• How intensively are search and filter functions used?
• Are there useful badges that can be displayed?
• Does it make sense to implement a quick-buy button?

Product detail page:

• Can I make my pictures more informative?
• Check the presentation of information, the content and the arrangement
• Which social proof elements trigger most with your target group?

Check-out

The check-out, the heart and motor of every online store. But be very careful here. We strongly advise against testing different layouts, arrangements or colors here. The check-outs of the most common shop systems are more or less sales psychology optimized.

In the check-out, it is other aspects and emotions that should inspire your A/B tests. Always ask yourself how you can guide the customers more effectively through the sales process or how you can increase customer trust through certain elements. 

PS: A/B testing in the check-out is complicated and requires a very sensitive approach. Adjustments in the Shopify check-out, for example, are only possible with Shopify Plus.

Navigation

The navigation is extremely important for every online shop. The navigation is the compass of your customers to navigate purposefully through the online store and to reach the desired products. Keep the following core aspects of a navigation always in mind as a starting point for an A/B test:

• Structure: Where does the search start? What basic aspects is it divided into? 
• Sense: How would a person outside the organization proceed? Would they need more or less of certain information? 
• Depth: How many levels make sense? Do more have to be added or can some be dispensed with? Are there points where complexity exceeds depth?
• Design: Can we make the menu more playful and intuitive? Are there elements or images that can help here?
• Copy: Which words trigger in the menu guidance? 

Search

In the past, we have noticed a 3-7x higher Conversion Rate in online shop sessions with a search query than in sessions without. Therefore, it can always make sense to check the design, layout and placement of the search function in an A/B test.

New Features

Unfortunately, it still regularly happens that online shop operators implement new functions directly without having previously checked their actual effect. It is usually a notion of unity that drives here. Many assume that best practices in other online shops are universally valid. That is not the case.

An A/B test always gives you the necessary decision base before the implementation of new features to check whether this would really be in favor of the target group.

What are the Advantages of A/B Testing in E-commerce? Why is it Important?

The biggest advantage of A/B testing is making decisions and improvements based on a data foundation, rather than personal preference. This is particularly noticeable in four areas, which we would like to introduce in this section.

Understand Customer Journey and Customers

Every improvement or change should be in the interest of the target group and the customers. Every A/B test provides you with information about what is really important for these people and what emotions work in the end. So with every A/B test, you get to know your target group more deeply and start to understand their purchasing motives and intentions in the long term and improve their Customer Journey. In the end, the bait should always taste good to the fish and not the fisherman.

More Revenue and Profit

In the best case scenario, A/B tests also pay off monetarily. Every A/B test usually checks at least one KPI. The result is reflected in an improvement or deterioration.

For example, an increased Conversion Rate leads to more orders, which is noticeable in the revenue. If you manage to increase your Average-Order-Value through targeted A/B tests, your profitability and profit will inevitably increase as well because package-related costs decrease.

Independence from Advertising Costs

In the past two years, advertising costs have risen significantly. In 2019, it was still comparatively cheap to buy new customers and impressions, but that's not the case anymore. Inevitably, online shop operators have to deal with how they can get even more out of their advertising budget and the existing visitors. And this is where Conversion Rate Optimization and A/B testing come into play. Through targeted A/B tests, more visitors are converted into customers and specific KPIs like revenue and profit increase. Completely detached from the advertising budget.

Understand and Utilize Data

Every visitor to your online store leaves data with every click. And you should definitely start understanding and using this data. In a world where data has taken on the relevance of raw materials, it is absolutely necessary to use it.

The goal should always be to improve your online store for your customers in the long term. Every data point can help you with this and give you indications for possible improvements. For instance, bounce rates in your analytics tools provide clues for basic problems or heatmaps show movements in your online store. You should always try to use data consistently and not treat it stepmotherly.

How Can the Success of A/B Testing in the Online Shop be Measured?

→ Success parameters/metrics, which can measure the potential success of A/B testing in e-commerce.

In each A/B test, it is necessary to verify a specific KPI. It is important to understand that the success can only be measured based on the selected KPI. For example, you cannot assume that a test to increase the Conversion Rate will also positively affect the Average-Order-Value.

Conversion Rate

The Conversion Rate is the ratio of the total number of shop visitors to the total number of orders in a specific period (day, week, month …). A successful A/B test causes a positive change in Conversion Rate and results in a higher number of orders. This, of course, also has an impact on the revenue.

Average-Order-Value (AOV)

Every online shop operator dreams of a higher Average-Order-Value (AOV). Turning this dream into reality is possible with an A/B testing strategy. Therefore, the success of an A/B test can also be measured in terms of a higher Average-Order-Value.

PS: For A/B tests for a higher AOV, focus on the issues of cross and upselling as well as bundling.

Average-Revenue-per-User (ARPU)

The Average-Revenue-per-User is probably the most important metric in A/B testing, as it combines the two metrics AOV and CR. If the AOV rises, but the CR falls, the ARPU stays the same. If both values rise slightly, the ARPU rises noticeably. We usually evaluate our tests focusing on the ARPU.

Individual Goals

In addition to these clearly defined KPIs, you can of course also set your own individual goals. For example, you can focus solely on a deeper understanding of your target group and neglect the other KPIs. If you choose such an approach, you should definitely focus your A/B tests on this goal and not pursue other goals.

What Tools are Suitable for A/B Testing in E-commerce?

Luckily, we don't have a shortage of excellent A/B testing tools and there is a wide range of tools available. As is often the case, the question of the right tool is answered with the requirements you have for such a tool. Therefore, there isn't one tool that suits everyone. We always recommend talking to as many different providers as possible and making a decision afterwards. We had good experiences with ABlyft in Kameleoon and VWO Testing. There is also an article in the Content Hub introducing the seven best A/B testing tools..

Conclusion on A/B Testing in E-commerce

A/B testing in e-commerce offers a lot of possibilities for counteracting current market trends, like increased advertising costs, and for improving your online store for your target group in the long term. To ensure that your A/B testing experiments end up being successful, you should always approach the topic with a clear strategy and a solid procedure. Be clear about what you want to achieve with your A/B tests and you will definitely take off!