Everything you want to know about SEO A/B testing

Have you heard of A/B testing in SEO and want to see whether it's something for your website? Then this article is a good place to start. Here you'll read everything you need to know about SEO A/B testing.

What types of A/B testing are there?

Within SEO, you can run A/B tests in two ways: with SEO split tests or with time-based tests. Let me explain the difference.

What is SEO time-based testing?

With time-based testing, you test a hypothesis on one and the same page. You make improvements and compare the results from before and after those changes. This approach is tricky, because a page never exists in isolation. External factors can strongly influence the outcome — think of seasonal effects such as the summer period, when results are often lower anyway. That makes it hard to determine exactly what effect your changes have had. Other external factors can play a role too, such as Google core updates, competitors starting on SEO optimisations, or new ads in Google.

Time-based SEO testing isn't the most reliable way to run an SEO test, but sometimes you have no other option. Time-based testing suits smaller websites that don't have enough pages or visitors to run a split test. It can also be handy if you want to see results quickly and don't have time to set up a split test.

What is SEO A/B split testing?

A/B split testing is best known from the world of conversion rate optimisation (CRO). There, you test two versions of a page — version A and version B — with different groups of visitors over a set period. Within SEO, however, A/B testing works slightly differently.

In SEO, you can't simply create two almost identical pages with just one difference. Google regards such pages as duplicate content, so usually only one version stands a good chance of ranking. SEO A/B testing therefore calls for a different approach.

This method is only suitable for larger websites with enough organic traffic. Does your site still get little SEO traffic? Then it's wiser to work on growing your organic reach first, or to use time-based testing.

Do you have several pages that rank well and attract a steady number of visitors from search engines? Then you could consider SEO A/B split testing. For a reliable test, you need a group of comparable pages. Think of pages with the same goal, a similar layout and content structure. When selecting and analysing these pages, pay attention to the following:

• Choose pages with a similar search intent (informational, navigational, commercial or transactional).
• Choose pages with a similar layout (blog page, product page, category page).
• Watch out for seasonal effects; choose pages that peak or dip in the same season.
• Choose pages with roughly the same number of monthly organic visitors and impressions.
• Make sure the pages get enough traffic and impressions to draw sound conclusions.
• Test only 1 hypothesis/element at a time.
• Think ahead about the effect you want. 
• To measure the impact as accurately as possible, it helps if the pages currently rank at a similar level.

With the selected group of comparable pages, you can now set up an SEO A/B split test. For such a test you usually need two groups: a test group and a control group. So you split the collected pages in two. In the test group, you make the changes that follow from your hypothesis. You leave the control group untouched — it acts as a reference point, so you can measure the impact of the changes properly.

What can you test with an SEO A/B test?

You can use SEO A/B tests for all kinds of optimisations. Think of content changes to key elements such as the page title, H1 or subheadings, but also technical changes such as structured data or adjusting the meta description. The following elements lend themselves well to an SEO A/B test:

• Page title
• H1 heading
• H2 and H3 subheadings
• Existing structured data scripts
• Video/image alt tags and/or file names
• Adding/removing images/videos
• Meta description (not a ranking factor, but can affect CTR)
• With/without publication date (and/or updating the publication date - possible effect on CTR)
• Content improvements
• Longer and shorter copy
• New internal links
• New backlinks
• Improved page speed
• Changes to the page design

Why SEO A/B testing?

Because you'd rather base your changes on data than on gut feeling. With SEO A/B testing, you collect data to base your improvements on, and you have direct proof that your changes really are improvements.

The pros and cons of SEO A/B testing

The main advantage of SEO A/B testing is that you know for sure that changes really work, and that you can better predict the results of improvements. SEO A/B testing has its drawbacks too. Depending on the test method you choose, you'll face different pros and cons.

Time-based SEO testing

The biggest drawback of time-based SEO testing is that you can't rule out external influences. Because your timeframe differs, you never know 100% for sure whether the results are due to your test or to outside influences. If you want to be certain your hypothesis is right, a time-based SEO test is less reliable than a split test. Time-based testing is, however, suitable for smaller websites, because you don't need large groups of similar pages and it doesn't matter if you have few visitors. Time-based testing also works well if you want to see results quickly.

A/B split testing

The biggest drawback of split testing is that it takes a lot of time. You test first, before actually rolling out the improvements to all pages. As a result, you see results later than if you made changes without testing, or if you started with time-based testing. You also need a minimum number of visitors to draw sound conclusions. Another drawback is that tested changes may work for some pages and not for others, even if you've put the test groups together carefully. Not every page or keyword gives the same results. On the other hand, A/B split testing does give you accurate results, and you can roll out improvements straight away across a large group of pages. That delivers more extra traffic than when only a few pages are involved.

Despite its limitations, SEO A/B testing can deliver valuable insights. You can use those insights to support future decisions. Do stay critical, though: past success is no guarantee of future success.

How can I best monitor my SEO A/B tests?

There are now various tools available for setting up and monitoring SEO A/B tests. But if you're just starting with SEO testing and aren't yet sure what it will deliver, such a tool can be too big an investment. Fortunately, with data from Google Search Console (GSC) and some basic knowledge of Excel or Google Sheets, you can get surprisingly far.

Depending on the goal you want to achieve, you'll probably want to monitor other metrics as well. For example:

• Improving Google rankings (positions & clicks)
• Improving CTR from Google (CTR, impressions & clicks)
• Increasing organic traffic (impressions & clicks)
• Improving mobile usability (impressions, clicks, CTR & positions for mobile devices)
• Increasing the number of rich results (external data on rich results from, for example, SEMRUSH)

Start tracking all relevant metrics as soon as you start the test, for example in Excel or Google Sheets. Compare the results of the test group with those of the control group, or with the data from the period before the changes. Ideally, the test group's figures will be visibly ahead of those of the control group or the earlier timeframe. That's a clear sign your test has turned out positive and has therefore succeeded.

What can I do with the results of an SEO A/B test?

If you've run a good SEO A/B test, three conclusions are possible based on the hypothesis and your results:

1. The hypothesis is wrong: the results have (significantly) worsened
   • Have the metrics you monitored during the test period worsened across the entire test group? Then your hypothesis is probably wrong. In that case, it's best to reverse the changes and give the pages a few weeks to recover their 'old position'. You can then set up a new test based on a new hypothesis.
2. The hypothesis may be right: the results haven't changed much
   • Have the metrics you monitored during the test period not improved or worsened significantly? Or do the results differ too much per page? Then you can't draw a sound conclusion. In that case, it's best to reverse the changes and give the pages a few weeks to recover their 'old position'. You can then set up a new test based on a new hypothesis.
3. The hypothesis is right: the results have (significantly) improved
   • Have the metrics you monitored during the test period improved? Then you've achieved a great result! In this case, you'll want to roll out the improvement to the pages in the control group as well. If you like, you can set up a new test for another element on the page to improve the pages even further.

So, depending on your conclusion, you either reverse the changes within the test group or roll them out within the control group. Above all, use your common sense to assess the trends within your test and control groups: is it coincidence, or is there a real cause-and-effect relationship?

SEO A/B test step-by-step plan

Is an SEO A/B test useful and relevant in your situation? And do you want to start SEO testing as soon as possible? Then our SEO A/B testing step-by-step plan will set you on your way!

1. Formulate a hypothesis

You start with a hypothesis in which you describe the expected result as clearly as possible. First decide exactly what you're going to test, and set your hypothesis on that basis. An example:

You expect the CTR of your pages to go up if you include figures and data in your meta titles wherever possible. You could then phrase your hypothesis as follows: By including figures in the meta titles, more people will click on the search result and the CTR of those pages will rise. Your SEO A/B test will then test this hypothesis.

Also decide which type of test is most suitable.

Time-based testing vs A/B split testing 

• Small website with few pages?
• Then time-based testing is the logical choice.
• Large site with many similar pages? Then choose A/B split testing. 
• Little organic traffic? Then time-based testing is more practical.
• Lots of organic traffic? Then A/B split testing pays off. 
• Need results fast? Go for time-based testing.  
• Have more time? Then you could consider A/B split testing. 

2. Choose your page(s)

First decide which pages you want to test the hypothesis on. In the example above, the hypothesis probably fits best with a specific group of research pages or blog articles. So, depending on your hypothesis, you choose which pages to include in the test group. 

Going for an A/B split test? Then select a set of comparable pages based on the right criteria. Working with a time-based test? Then choose one specific page on which you make the changes and compare the results over time. 

SEO time-based testing vs SEO A/B split testing

• Working with one page? Make sure it has a clear search intent and a clear goal. Testing with several pages? Then choose pages with a similar intent and the same goal. 
• For a time-based test, it's smart to choose a page that isn't seasonal. Running an A/B test? Then it's important that all pages peak or dip in the same season. 
• For a time-based test, watch out for Google ads in the search results. Preferably choose a page where they are limited. This matters less for A/B testing — there, it's all about comparing the pages with each other. 
• For both methods: choose pages, or a page, that already attract some organic visitors. Without enough traffic, the results are hard to interpret. 
• Want to see an effect quickly in a time-based test? Then a page that already ranks reasonably well works best. In a split test, it's important that all pages rank at about the same level, so you can compare fairly. 

3. Randomly assign your test group and control group (for A/B split testing)

Before you start testing, you still need to define your control group and test group. Ideally you do this randomly, so no preference or unconscious pattern creeps in. In Google Sheets, you can easily sort your pages randomly. Turn the selection above into 2 groups: 1 test group and 1 control group. You'll make the changes in the test group and do nothing in the control group.

4. Decide on the changes

Now that you've defined the test group or page, you can decide on the exact changes. Keep in mind that you only want to test 1 variable, so change, for example, either the title or the description, but not both. That way it's clear afterwards which change led to which result. 

So for each page, decide what the new situation will be once the change is made. Say you want to include a figure in every title: at this step, decide what the new title, including the figure, should be. You do this only for the pages in the test group. The pages in the control group stay unchanged. 

5. Set the test period (2 - 6 weeks)

How long do you want the SEO A/B test to run? That depends largely on what you're going to test. Raising rankings generally takes more time than raising CTR. Small changes to a meta title or description are also generally picked up faster than a complete improvement of the content on the page. 

To be on the safe side, it's best to stick to a test period of 6 weeks. Within that time, you can assume Google has picked up all the changes and that you can actually see results.

6. Make the changes on the test pages

You've set all the conditions for your SEO A/B split test. Time to start the test! To do so, you prepare the changes. Make sure you don't change more than you decided beforehand, because you only want to test 1 variable. Make the changes only on the pages within the test group.

7. Submit an indexing request through Google Search Console

To make sure Google picks up the changes as quickly as possible, you can resubmit the test pages for indexing in Google Search Console. It doesn't guarantee that Google will pick up the changes, but it does increase the chances. You submit such a request by entering the URL in the URL Inspection tool and then clicking 'Request indexing'.

8. Start testing

Everything is ready, so your SEO A/B test can begin! From now on, the test period is running. Make a careful note of when the test started and when it ends. Also make sure that no other changes are made to the test or control pages during that period, because that would make your test results unreliable. 

Does Google roll out a core update during the test period? That can also distort your test results. Got a negative test result? Then don't immediately assume it's caused by your changes; it could also be the result of the Google update. In that case, build in a 2-week pause (after the Google update has fully rolled out) and restart your test. 

9. Collect data and monitor

You want to measure the effects of your changes, so it's important to collect data. We've already explained how best to monitor and analyse results. It's smart to set up a dashboard and keep track of the data during the test period. If something goes completely wrong, such as a sharp drop, you can step in in time. If you see growth instead, great, but try to curb your enthusiasm a little and wait for the full test period. After all, you don't want your conclusions to be based on chance.

10. Draw data-based conclusions

You have all the data, so put it to use! Not seeing much difference in the results, but your gut says the changes are an improvement? Then do NOTHING with that feeling. You have no guarantee that your changes have a positive effect, and you don't want to risk seeing declines in the long run.

So if you see no clear difference between the start and the end of your test, either positive or negative, don't roll out the changes to your control group (or other pages). Only when you see clear results can you label them as effective and roll them out.

11. Apply your conclusion to the test and/or control pages

Seeing a positive result from your changes? Then you can roll out the changes across your entire test set (both test and control pages). Be careful about making the same changes to pages outside your test set. The changes may turn out positive there too, but they may not. If you want to be sure, set up a new test.

Have we sparked your interest in SEO A/B testing? We're happy to help you set up your SEO testing strategy! Get in touch with us so we can look together at what Partout can do for you.