A/B testing for Shopify: how to decide what to test when you don't have much traffic
A/B testing compares two versions of a page by sending each to a share of your visitors, and for most small Shopify stores the hard part is not running the test but having enough traffic to read it. If your page gets a few thousand visits a month, a test on a small change can take longer than the change will matter. This guide shows the maths with an illustrative example, what Shopify offers, and how to decide what to test, and what to simply fix, when traffic is thin.
In the video, a store loses to a stronger competitor. Most of the time that happens because of problems you can already see, not because nobody tested a button color. So the order is: find the likeliest problems from evidence, test the ones you can, fix the rest, and keep the store you are beating in view.
Want a list of candidates before you plan a test? Find what to test first on your store →
What is A/B testing, and what is it not?
In an A/B test, visitors are split at random between the current page (A) and one changed page (B). Because both run at the same time, seasonality, ad spend and press hit both equally, so a difference in conversion can be credited to the change. That is what separates it from changing the page and comparing this week with last week: a before-and-after comparison cannot tell your change from everything else that moved. It is still useful, but it is evidence of lower quality, and you should say so when you report it.
How much traffic does an A/B test need?
The number of visitors depends on four things, which is exactly what Evan Miller's sample size calculator asks for: your baseline conversion rate, the smallest effect you want to be able to detect, the significance level (how often you will accept a false alarm) and the statistical power (how often a real effect of that size will be caught). The smaller the effect and the lower the baseline, the more visitors you need, and the relationship is steep.
Here is an illustration. Take a baseline conversion rate of 1.4%, the median for Shopify stores in Littledata's October 2026 benchmark (433 stores, 90 days). With a 5% significance level and 80% power, we calculate that:
Those are visitors per version, so double them for the test. Detecting a 20% relative lift (1.4% to about 1.7%) takes roughly 60,000 visitors in total. A product page that gets 5,000 visits a month would need about a year. Detecting a 50% lift takes roughly 11,000 visitors, which is far more realistic, but a 50% lift is a big change, not a button color.
Two cautions on these figures. They are our own calculation for illustration, with standard settings; calculators differ in their defaults (one-sided versus two-sided tests, power), so run yours with your own baseline instead of trusting ours. And the baseline matters: if your page converts at 3%, the same lift needs fewer visitors, and a page with 300 visits a month needs a very different plan from one with 30,000.
What goes wrong when you test with too little traffic?
- Peeking. Checking the result every day and stopping when it looks significant. Miller's How Not To Run an A/B Test shows that repeated looks inflate false positives: in his worst-case logo example (50% baseline, a significance test after every observation, stopping at 150 observations), stopping at the first significant result gives a 26.1% false-positive rate with no real effect, and ten looks turn a reported 1% significance into roughly 5%. Fix the sample size before you start and do not stop early.
- Declaring a winner you cannot defend. A "lift" of 12% from 40 orders can be noise. If the test did not reach its planned sample, the honest result is "no conclusion".
- Testing something tiny. The smaller the effect, the bigger the sample. Button color and one-word copy tweaks are the usual way to waste a month.
- Splitting thin traffic across many pages. Concentrate on the page that carries the most visitors and sits closest to the money.
- Changing something else mid-test. A new ad campaign, a sale or a theme update during the test breaks the comparison.
What does Shopify offer for A/B testing?
Shopify's changelog describes Rollouts (posted 5 June 2026): you can schedule a new theme or checkout, release it gradually to a percentage of visitors, and run an A/B test between two different themes or checkout setups. A copy of your published theme is created so you can keep editing the live version, and several mutually exclusive experiments can run at once. As described there, it tests whole themes or configurations; the post states no plan requirement, so check your own admin. For single-element tests on product pages there are third-party apps in the Shopify App Store; read each listing's current description and pricing yourself, since we have not tested them. Whichever you use, the maths above applies: the tool splits the traffic, it does not create it.
Without any tool, Shopify's reports still let you compare periods. Our guide to the good conversion rate for Shopify explains which numbers to record first.
What should you do instead when traffic is low?
- Look before you test. Session recordings and heatmaps show where visitors hesitate. Microsoft Clarity offers both; a handful of recordings often shows an obvious problem that needs no test.
- Fix obvious problems on evidence. A confusing headline, a missing shipping policy or a broken mobile button does not need a split test; it needs a fix. Our Shopify CRO audit checklist lists them.
- Test only big changes. A new offer, a rebuilt product page or a different page layout can produce a large enough effect to read on modest traffic.
- Compare equal periods and log everything. If you ship a change without a split, record the date, compare equal periods from the same weekdays and note any ad or price change. Call the result what it is: a before-and-after, not a proven lift.
- Ask people. Five short conversations with recent buyers or a one-question survey can tell you what nearly stopped them.
How do you choose what to test first?
Write each idea as a hypothesis: "Because [evidence], changing [element] to [version] will increase [metric]." Then rank by three questions (our judgment, not a standard): how strong is the evidence, how large a change would it be, and how much traffic does the page get. A strong-evidence, high-traffic, big-change idea goes first. Our guide on how to increase your Shopify conversion rate groups candidates by stage of the visit, and the Shopify product page guide covers the page where most tests happen.
Evidence can come from your analytics, from recordings, from customers and from comparing your store with the one that beats you in your niche. That last source is the one small stores use least: open the leader's product page next to yours and write down every difference a stranger would notice in five seconds.
Can an audit tell you what to test?
It can supply candidates, not verdicts. A tool that reads your page (EliteVault, ours — full disclosure) can flag the problems a stranger would see, which makes it a hypothesis source. It cannot see your analytics, it reads a screenshot, and it does not run or judge experiments.
Where test ideas can come from: what audits of real stores flag
A test is only as good as its hypothesis, and a hypothesis is only as good as the evidence behind it. Across 134 audited stores, the fixes the audits ranked highest mentioned these themes. Each row is a starting hypothesis, not a result:
| Theme in the audit's top fixes | Share of stores | A testable hypothesis it suggests |
|---|---|---|
| Hero and offer clarity | 67.9% | A headline that states what you sell and for whom beats the current one |
| Trust and social proof | 64.9% | Real reviews near the buy button lift add-to-cart |
| Buy button and path to purchase | 49.3% | A single obvious primary action beats two competing ones |
| Product imagery | 40.3% | In-use and in-scale photos beat studio-only shots |
Read it with some suspicion: the stores picked themselves, the audit reads a screenshot (so it cannot see your analytics) and the themes are a keyword match on the audit's own fix titles and explanations. The way to use it is as a source of candidates. Run your own store through the audit, take its first fix and write it down as a hypothesis with a metric. For example, the stored audit we use in our guides flagged two blockers on a home-decor store: a fake-looking countdown timer and a lack of authentic social proof. Neither needs an A/B test to justify; one is a trust problem you can simply remove, the other is content you collect and place. Both come from evidence on the page, not a hunch.
What EliteVault does not do: it does not split your traffic, run experiments or report test winners. Its job here is to help you decide what to test and in what order.
See the problems an audit flags on your own page: Get the first fix to test on your store →
On plans: anonymous visitors and Free accounts get the audit, the annotated screenshot, the ad-readiness verdict and the first fix. The other fixes with their how-to open with a one-time $7 unlock for that audit, or with Pro ($19 a month) for every audit. Use the first fix as your first hypothesis; unlock the rest only if you want the full list to plan a longer test queue.
A simple test plan you can copy
- Hypothesis: one sentence with the evidence.
- Metric: one primary metric, such as purchase conversion rate or add-to-cart rate.
- Sample: compute it before you start with your baseline and a lift you would act on; write down the date you will stop.
- Change: one thing, on a duplicate of the theme.
- Run: no peeking, no other changes, whole weeks.
- Decide: act only if the planned sample is reached; otherwise record "no conclusion" and pick the next idea.
Use the evidence to beat the store next to you
You do not need a testing programme to improve a small store; you need to stop guessing. Find the biggest gap between your page and the leader's, fix what the evidence supports, and keep tests for the changes big enough to read. For tools around that, see website audit tools, and for the page itself, our Shopify store audit.
Frequently asked questions
How much traffic do you need for A/B testing?
It depends on your baseline conversion rate and the smallest lift you want to detect. As an illustration, at a 1.4% baseline, detecting a 20% relative lift needs roughly 30,000 visitors per version with standard settings, while a 50% lift needs roughly 5,500. Use a sample size calculator with your own numbers.
Can you do A/B testing on Shopify?
Yes. Shopify's changelog describes Rollouts, which can run an A/B test between two themes or checkout setups and release changes gradually. Third-party apps exist for single-element tests. Check your own admin for plan availability and read each app's listing.
How long should an A/B test run?
Until it reaches the sample size you calculated before starting, rounded up to whole weeks. Stopping when the result first looks significant inflates false positives.
What should you test first on a small Shopify store?
Large changes backed by evidence, on the page with the most traffic closest to checkout: the offer and headline, the product page layout or the proof near the buy button. Fix obvious problems without a test.
Does EliteVault run A/B tests?
No. EliteVault audits a page and flags problems, which you can use as hypotheses; it does not split traffic or report test winners.