SEO A/B Testing: How to Evaluate Your SEO Measures

1. What is SEO A/B Testing?
2. Why is A/B Testing useful in SEO?
3. Who benefits from SEO A/B testing?
4. What types of SEO A/B testing are there in practice?
5. What details are important for an SEO A/B test?
6. How does an SEO A/B test work?
7. How to set up your SEO A/B test
8. Create Test & Control Group with ChatGPT
9. Test Evaluation: SEO A/B Testing with Causal Impact
10. Conclusion: Should you invest your time in SEO split tests?

In search engine optimization, it is sometimes difficult to make the impact of measures measurable. With SEO A/B testing, we create exactly this possibility.

What is SEO A/B Testing?

In an SEO A/B test, the impact of a change on your website is analyzed based on certain SEO KPIs. Compared to a classic A/B test, where two variants of a page are created, here multiple pages are divided into two groups. So it's a split test. With correct division, seasonal fluctuations and various updates affect test and control groups and are thus neutralized. 

Why is A/B Testing useful in SEO?

With SEO A/B testing, you can substantiate your SEO measures with numbers. Measures can not only be better evaluated, but the internal value of SEO projects can also be increased. Through split testing, we can better predict the output of a project, which facilitates resource planning and helps convince decision-makers. Another important point is to identify negative effects – regardless of how long a person has been working in SEO, not every idea brings a positive effect. If a measure is tested before implementation, changes can be undone if the desired effect does not occur. 

SEO test procedures can measure both small and large changes on your website. You will quickly notice that sometimes small changes can have a big impact. Everything is possible from technical adjustments to content changes, as long as your website meets the requirements. 

Who benefits from SEO A/B testing?

This type of testing is aimed at larger websites, which meet the following requirements: 

• At least 1,000 organic clicks per day across test & control group
• The total population should consist of at least 300 URLs
• The pages within the test must show the same template
• You must be able to make the desired changes on your website

Click and page count are benchmarks. That means you can also conduct tests if your site does not fully meet these guidelines. However, the probability of a significant result decreases. It is also important that you have an analytical understanding, can prepare the figures and can interpret the results after the test has ended. 

What types of SEO A/B testing are there in practice?

In addition to a split test, you can also conduct a time-based test in SEO. In a time-based SEO test, you make the changes to a group of pages and wait for a certain period. Then you look at the change in organic traffic between the control period (period before the change) and the test period. With this type of test, there is no test & control group. This testing option is especially suitable for small websites that do not meet all the requirements. Here it should be noted that with a simple comparison, no significance calculation takes place. 

With a statistical SEO split test, in general, you can draw conclusions from the data within 2 to 6 weeks. If your website meets all the requirements, Google offers the right concept here: With Causal Impact, the change of a certain KPI is tested based on your test change against an unchanged control group.

What details are important for an SEO A/B test?

The most important thing is the data basis. If we start with faulty or incomplete data, we cannot expect a correct result. Therefore, take enough time to prepare your data. 

How does an SEO A/B test work?

First, you need to know what measures you want to implement next. An SEO strategy is therefore a prerequisite for conducting tests. It is not recommended to conduct an SEO test for the sake of testing. Rather, split tests should support your work.

The process can then look like this: 

1. Determine testcase
2. Form hypotheses: Draw up the null & alternative hypothesis
3. Build the total population: All pages that you need for your test
4. Sample Size Calculation: Determine the number of URLs in the test group
5. Form test & control group
6. Make changes
7. Wait 2-6 weeks
8. Evaluate test

How to set up your SEO A/B test

First, you must be clear about what you want to test and what result you expect. To do this, you create a H0 (null hypothesis) and an H1 (alternative hypothesis). These must mutually exclude each other:

H0 = The integration of emojis in my snippets has no impact on the user and does not influence clicks.

H1 = The integration of emojis in my snippets has an impact on the user and positively influences clicks.

The H0 is the initial hypothesis and is valid until the opposite is proven using data. In order for you to be able to create the test and control group, you need a total population of URLs that are suitable for your test. As a data base, you export the organic clicks per day per URL for the last 3 months and save them in an Excel file. Whereby 3 months are the minimum here, you can also refer to a longer period. 

When dividing the URLs, you can always make a 50:50 split. If you want to test the change on fewer URLs, you can calculate with a "Sample Size Calculation", how many URLs of the total population belong to your test group. 

Create Test & Control Group with ChatGPT

The Advanced Data Analysis of ChatGPT can help you divide your URLs into test and control groups: To do this, upload your prepared Excel file in ChatGPT and first have the data analyzed.

If the content is interpreted correctly, you can divide the URLs into two groups. You can use this prompt for this: "Can you help me rearrange the data differently? My goal is to divide the URLs into two approximately equally sized groups and calculate their correlation. The correlation should be over 0.9." You should aim for a correlation of at least 90% - the higher, the better.

It may be that chatGPT does not immediately find an optimal solution and suggests a different approach. My experience has shown that an optimal result occurs after 2-3 rounds. 

You can now output the data with a simple prompt in an Excel file: "Thanks! Can you output the URLs with their corresponding group in an Excel table for me?"

Do you want to check the results of ChatGPT? With the Excel function "KORREL" you can calculate the correlation of two ranges.

After you have determined test and control groups, it is important that you make your changes collectively within one week. If you need longer for your change, it is recommended to enrich the total population with the new traffic data and check the correlation again.

With the going live of the measure, the SEO test begins. If you want to be absolutely sure that Google sees all the changes, you can manually trigger the crawl.

You should now let your test run for at least two to six weeks to get a meaningful result. If you're curious, you can also take an initial look at the numbers after a week. However, I recommend that you let it run for at least 2 weeks. 

Test Evaluation: SEO A/B Testing with Causal Impact

For the test evaluation, add the traffic data (click per URL per day) in your Excel file for the test period. Then sum up the traffic data per group and save these in a separate CSV file. This should look as follows: 

Y = Test Group (Sum of Clicks)

X = Control Group (Sum of Clicks)

The test evaluation then takes place in RStudio. The model that evaluates your test is Causal Impact. Don't worry, you don't have to program yourself - you can find the complete code on Github. 

After you have downloaded all relevant packages (Causal Impact: Be sure to select "install dependencies", bsts, BoomSpikeSlab, Boom & readr), you transfer the code to RStudio. Now you just have to specify the path of your CSV file and set the pre and test period in the form of line numbers. In the following example, the pre-period is from line 1 to 150 and the test period is from line 151 to 180.

This should look like this: 

After you have provided all the information, you can execute the program. You automatically get a report and a graphic result, without having to calculate anything yourself.

On the left you will find the report. The evaluation tells us how the traffic has changed compared to the forecast and whether the result is statistically significant or not. 

The graphic that is output is divided into three areas:

• Original: The first field shows through the blue line a prediction of the traffic trend without change. The black line shows the actual course of the data.
• Pointwise: This view shows the difference between observed data and the estimate.
• Cumulative Impact: Here the effect is cumulatively shown. 

Which tools are suitable for SEO A/B tests?If you want to check your SEO measures using SEO A/B testing, you can set up a free testing environment using RStudio or fall back on one of the following tools: 

• Ryte

• Semrush

• SearchPilot

Conclusion: Should you invest your time in SEO split tests?

SEO split tests can help you measure your SEO success. They help you (and your stakeholders) understand the value of your SEO measures so that you can make informed decisions and design your SEO strategy successfully in the long term. Also, SEO is and remains unpredictable: What works on one website does not necessarily have to be advantageous on another website or could not be relevant tomorrow. Through tests, you can find out what works best for your website and proactively react to changes. If your website meets the requirements, you should from now on check your SEO measures using SEO A/B testing.