How to A/B test your Shopify store (without fooling yourself)

What an A/B test is

An A/B test (or split test) is a controlled experiment: your visitors are randomly split into two groups, one sees your current page (A, the control) and the other sees a new version (B, the variant). Whichever earns more wins.

The power is in the word controlled. Both groups shop at the same time, see the same ads, get the same offer. So when one side earns more, the page is the most likely reason, not the weather, a celebrity post or a payday weekend.

Compare that with the usual approach: change the product page on Monday, check sales on Friday, and decide it "worked". You have no idea what else changed that week. A proper split test on Shopify removes the guesswork.

The metric that matters: revenue per visitor

Judge every test on revenue per visitor (RPV). It's simple:

Conversion rate on its own is a trap. Here's a hypothetical example to show why. Say each version gets 1,000 visitors:

Version B "wins" on conversion rate and loses on money. And that's before margin: the discounted orders also earn less profit each. RPV catches this. Discounts can't fake it.

RPV also gives credit to changes that grow the basket, like bundles, quantity breaks and smart upsells. If you're working on those, our guides to increasing average order value on Shopify and Shopify bundles and upsells are good test fodder.

What to test first

Start with big changes on your highest-traffic pages. More traffic means faster answers, and bigger changes produce differences you can actually detect.

1. Your busiest pages

Check your Shopify analytics for the pages that get the most sessions, usually the home page, your hero product page or the landing page your ads point to. A Shopify landing page test on a page that gets a trickle of visitors will take forever to call.

2. Offer presentation

How the offer is framed often matters more than any single design tweak. Test things like how bundles are shown, how pricing and savings are displayed, what's included, and how shipping and returns are explained. Keep the actual offer the same in both versions so you're testing presentation, not price.

3. Page structure

The order and depth of the page: what shows above the fold, where reviews sit, how quickly the "why buy" story lands, how the page handles objections. Rebuilding the structure of a page is a big swing, and big swings are what you want early.

4. Then the small stuff

Button colours, headline wording and icon styles come last. They can matter, but the lifts tend to be small, and small lifts need lots of traffic to prove.

Impact ratings are rules of thumb, not promises. Every store is different, which is exactly why you test.

How to A/B test on Shopify, step by step

Follow these ten steps in order. Skipping any of them is how good stores end up rolling out changes that quietly cost them money.

1. Write a hypothesis. "Showing the 3-pack bundle first will raise average order value because most customers reorder within a month." A clear hypothesis tells you what to change and what to learn, win or lose.
2. Pick one primary metric. Revenue per visitor. You can watch conversion rate, AOV and add-to-cart rate too, but decide in advance which number crowns the winner.
3. Split traffic 50/50. Random, even, and sticky, so a returning visitor sees the same version each time.
4. Keep everything else the same. Same offer, same prices, same ads, same email campaigns pointing at both. The page should be the only difference.
5. Decide sample size and duration up front. Use a sample size calculator, then commit to the number. No moving the goalposts later.
6. Run full weeks. People shop differently on a Tuesday than on a Sunday night. Run in whole-week blocks so both versions see the full cycle.
7. Don't peek and stop early. Results swing around wildly in the first few days. Stopping the moment B looks ahead is the most common way to crown a fake winner.
8. Call it. At your planned end date, check whether the difference is statistically significant, meaning it's unlikely to be random noise. Most testing tools report this for you.
9. Roll out the winner to 100%. If B wins, it becomes the new control. If A wins, keep it and move on.
10. Test the next lift. CRO testing is a habit, not a project. Each winner becomes the baseline for the next test.

How much traffic you need

It depends on how big a difference you're trying to detect. The smaller the expected lift, the more visitors you need to be confident it's real.

Think of it like flipping coins. If one coin lands heads 90% of the time, you'll spot it in a handful of flips. If it lands heads 52% of the time, you'll need a lot of flips before you can tell it apart from a fair coin. Test results work the same way.

What that means in practice:

• High-traffic stores can test smaller refinements and get answers quickly.
• Lower-traffic stores should test bigger changes, like a full page rebuild or a new offer presentation, and run each test longer.
• Everyone should use a sample size calculator before launching. Free ones are easy to find online. You enter your current conversion rate and the smallest lift you care about, and it tells you roughly how many visitors each version needs.

If the calculator says you'd need months, that's useful information. It's telling you to test something bolder.

Shopify A/B testing tools

There are three broad ways to run a split test on Shopify. The right one depends on what you're testing and how technical your setup is.

• Shopify-native and theme-based tests. Testing alternate templates or theme versions within Shopify itself. Good for page layout and content tests that live in your theme.
• Third-party A/B testing apps. Apps from the Shopify App Store that handle the traffic split, tracking and reporting for you. Many suit page, pricing-display or offer tests. Check carefully what each one can and can't test, and how it affects page speed.
• Server-side testing tools. Tests that decide which version to show before the page loads. More setup, usually needing a developer, but they avoid the "flicker" some front-end tools cause and suit more complex experiments.

Whichever you use, make sure it reports revenue, not just clicks or conversions, so you can judge on revenue per visitor.

Common mistakes that fool store owners

Most bad test results come from a handful of avoidable mistakes.

• Peeking. Checking daily and stopping when B pulls ahead. Early leads often vanish. Set the end date and stick to it.
• Too many variants. Testing A vs B vs C vs D splits your traffic four ways, so each version gets fewer visitors and the test takes far longer. Stick to one challenger at a time unless you have lots of traffic.
• Testing only during a sale. Sale shoppers aren't your normal shoppers. A page that wins during a sale may not win the rest of the year.
• The novelty effect. Returning customers sometimes click on something just because it's new. That bump can fade. Running full weeks, and longer for big changes, helps it wash out.
• Ignoring margin. A variant that pushes a low-margin product can win on revenue and lose on profit. Sanity-check winners against your margins before rolling out.
• Ignoring the mobile vs desktop split. Check your analytics: many stores see most sessions on mobile. Look at results by device: a change that helps desktop and hurts mobile can still lose overall. Design and QA the variant on a phone first.
• Changing other things mid-test. New ad creative, a big email send to one page, a price change. Anything that hits one version harder than the other spoils the result.

Tests don't stop at the page, either. Once a winner is live, your emails need to send people to it. Our guide to Klaviyo flows for Shopify covers bringing those customers back.