A/B split and multivariate testing

A/B testing, honestly

An A/B test shows half your visitors one version of a page and half another, then compares what happens. A multivariate test does the same with several changes at once. Done well, testing tells you what works for your own customers. Done without enough traffic, it produces confident-looking nonsense.

Many stores do not have enough traffic to test properly. If that is you, we will say so, and suggest better uses of your money.

Who it’s for, and when it isn’t

It suits stores with enough visitors and orders on a page for a test to reach a clear answer in a sensible time, and a specific idea worth testing.

It is not the right first step if:

• the page has few visitors or few orders. Most tests will not finish, or will give a misleading answer
• there are obvious problems still to fix. Slow pages, hidden delivery costs and broken checkout steps should be fixed, not tested
• your tracking does not match your real orders. We would start with GA4 setup and audits

What’s included

• An honest view of whether your store has the traffic to test, and which pages could support it
• A written hypothesis for every test: what we change, why and what we expect
• Test design, including what counts as success, how long it runs and when we stop
• Set-up, quality checks and monitoring in your chosen testing tool
• A report on every result, including the ones that showed nothing
• Alternatives to testing, where traffic is too low

How much traffic you need

There is no single number, and we will not invent one. Three things decide it:

• Your current conversion rate. The fewer orders a page produces, the longer a test takes. Orders matter more than visits.
• The size of the difference you want to detect. Spotting a small improvement needs a lot more traffic than spotting a large one.
• How many versions you test. Every extra version splits your visitors again. This is why multivariate tests need far more traffic than A/B tests.

Many small and mid-sized stores find that a test on a typical page would need to run for months. By then the season, the stock and the ads have all changed, and the answer means little. We work out the figure for your page before we start, and tell you if the answer is “not worth testing”.

If your store has low traffic

You still have good options, and for many stores they are better value than testing:

• Fix the obvious. Slow pages, hidden delivery costs, confusing variant choice and checkout errors do not need a test. Fix them. See conversion funnel optimisation.
• Use behaviour evidence. Heatmaps and recordings, customer service questions and reviews show what is putting people off.
• Make bigger changes. A test of a bold change can reach a clear answer on less traffic than a test of a button colour.
• Test where the traffic is. Paid landing pages are often busier than the rest of the store. See landing page optimisation.
• Compare before and after, with care. This is not a test. Seasonality, sales and ad changes all muddy it. We note what else changed and say how uncertain the result is.
• Ask customers. A short survey or a few conversations can answer a question a test never could.

How we run a test

1. Write the hypothesis. What is wrong, why we think so, what we will change and what we expect.
2. Fix the rules first. Success measure, sample size, run time and the people who will decide. We write these down before the test starts.
3. Change one thing at a time where we can, so the result means something.
4. Check it works. Every version is checked on phones and computers, and the tracking is checked against real events.
5. Leave it alone. We do not stop early because it looks like it is winning, and we run across whole weeks.
6. Record the result, including the tests that showed nothing. That is how you avoid repeating them.

Tools and Shopify

We choose a testing tool to suit your store and budget.

Shopify now has its own testing tool, Rollouts, which tests theme changes server-side, so there is no flicker and no extra script to slow pages down. Experiments need the Grow plan or above. It can’t test changes to theme code, and it has limited targeting, so some tests still need a specialist app.

Rollouts can test checkout and customer account settings, and reports the checkout conversion rate. On every plan you can brand the checkout and add apps to the thank-you and order status pages. Apps inside the checkout steps, and deeper changes, need Shopify Plus.

What it costs

Our pricing page explains how we charge. The price depends on how many tests are planned and how much build work each needs. The testing tool is a separate cost.

We start with an audit. After that, the work runs in sprints (plan, test or implement), each priced and agreed in writing before it starts. See the pricing page.

What we measure

• the main measure we agreed before the test, such as orders per visitor, and a few supporting ones
• revenue per visitor, so a rise in conversions does not hide a fall in order value
• whether the sample size and run time were reached
• how uncertain the result is, in plain words

We report every test, including the flat ones. We never describe an inconclusive test as a win.

Related

Testing sits alongside landing page optimisation, conversion funnel optimisation and GA4 setup and audits, which makes sure the numbers can be trusted. It is part of conversion rate optimisation.