A/B testing feels like rigor. On most pages it is procrastination with a dashboard. Here is how to know whether a test will teach you anything — and what to fix before you run one.
A/B testing has a halo. It sounds scientific, data-driven, grown-up. So when conversion is bad, the reflex is to spin up a test: green button vs. blue, this headline vs. that one, two-step form vs. one-step. Weeks pass. The dashboard fills with numbers. And nothing moves — because the variants were cosmetic and the page was broken in ways no button color can fix.
Here is the uncomfortable truth most CRO content skips: the vast majority of landing pages do not have enough traffic or conversions to run a valid A/B test at all. Running one anyway does not give you a wrong answer — it gives you no answer, dressed up as one. You stare at a 'lift' that is pure noise, ship the 'winner', and your real conversion rate does not budge.
This guide is the anti-hype version. Two questions, in order: (1) Is your page good enough that a test is worth running? (2) Do you have enough volume for a test to mean anything? If the answer to either is no, testing is the wrong move — and we will tell you what to do instead.
You do not need to memorize the statistics. You need one principle: detecting a real difference between two versions requires a sample size that grows fast as (a) your baseline conversion rate gets smaller and (b) the improvement you're hoping to detect gets smaller. A page converting at 2% trying to detect a modest relative lift needs far more visitors per variant than founders intuitively expect — frequently thousands of conversions per arm, not thousands of visitors.
Translate that to your reality. If your page gets a few hundred visits a week and converts in the low single digits, you are collecting a handful of conversions per week per variant. A test like that can run for months before it could possibly separate signal from chance — and by then your traffic source, season, and copy have all changed, so the experiment is contaminated anyway.
Don't trust a gut estimate here. Plug your real numbers into a public sample-size or significance calculator before you commit, and frame the output honestly: if it tells you the test needs more conversions than you'll plausibly collect in 4–6 weeks, the test is not viable yet. That is not a failure — it is the calculator doing its job.
Use a public sample-size calculator and put in your actual baseline rate and the smallest lift you'd act on. The number it returns is usually a reality check.
If you can't tick all four, you are not ready to test. Fix the gap first — these are the conditions that make a test capable of teaching you something.
When you lack volume, you have one reliable lever left: don't test cosmetic variants — make changes large enough that the effect is obvious. And the way to find those changes is not guessing. Most conversion problems on a low-traffic page are deterministic, not probabilistic: a missing CTA above the fold, a headline that names a category instead of a benefit, zero trust signals, a form asking for nine fields, an unanswered objection any honest founder already knows about.
You don't need 10,000 visitors to know a page has no social proof or that the hero never says who it's for. You need a structured read against a known rubric. That is the entire reason a deterministic audit exists: it inspects the page against fixed criteria and tells you the broken parts, today, with zero traffic required. Fix those, ship, and your before/after gap will dwarf anything a 2%-vs-2.1% test could ever surface.
We built our scoring around exactly this — a fixed set of dimensions checked the same way every time, so two people auditing the same page reach the same verdict.
If you want to see the criteria a deterministic audit checks, read the 12-dimension methodology — clarity, value prop, CTA, trust, social proof, objections, form friction, and more. None of it requires waiting on a test.
Walk these in order. The first 'no' tells you what to do before you touch an experiment.
No → Don't test. Rewrite the hero to make one specific claim about a specific outcome for a specific person. This single fix usually moves more than any button experiment.
No → Don't test. Cut competing actions down to one primary CTA. Demote the rest. You cannot interpret a test when intent is split four ways.
No → Don't test. Fix tracking first. A test on broken analytics is worse than no test — it produces confident, wrong conclusions you'll act on.
No → Don't test. Run a deterministic audit, fix every clear-cut issue, ship the whole batch at once, then re-measure your overall rate. Big, obvious wins don't need significance testing to be real.
No → Don't test. Keep improving deterministically and pour energy into traffic. Come back when the volume math works. Yes to everything above → Now a test will actually teach you something. Test one variable at a time.
Testing earns its keep once the fundamentals are solid and the volume is there. At that point it's the only honest way to settle close calls.
Enough weekly conversions that the calculator says a test can resolve in weeks, not quarters. High-traffic pages are where A/B testing pays for itself.
Clear value prop, one CTA, trust signals present, low form friction. You've exhausted the deterministic wins and now you're optimizing a page that already works.
Two genuinely good headlines, two real offer framings. When both candidates are credible and the gap is small, your judgment can't call it — only data can.
Pricing presentation, a new hero direction, a structural form change. Worth the rigor because the downside of guessing wrong is large and a test de-risks the rollout.
One important exception to 'you lack volume': some pages get plenty of traffic and still convert badly. That is not a sample-size problem — it's a fit-or-fundamentals problem, and it has its own diagnostic path. Don't jump straight to testing there either; first figure out whether you have a message-match leak, a wrong-audience traffic problem, or a page that's simply not making its case.
If that's your situation, the order of operations changes — and we've mapped it separately so you don't waste paid traffic running tests on the wrong layer.
Start with diagnosing high traffic, low conversion before you set up a single experiment.
A/B testing is a high-volume optimization tool. It is the last 10% of conversion work, not the first. If your page is leaking on fundamentals, a test will either give you noise or confirm something you could have fixed in an afternoon. Fix the deterministic problems first; reserve experiments for the close calls a healthy page actually faces.
The fastest way to know which camp you're in is to run a structured read of your page. Get the full picture in our deep-dive on the broader process, and look at a real example of what a clinical audit surfaces before you decide anything.
Paste your URL into the free mini-audit and get the top fixes in about 60 seconds. If they're fundamentals — fix them, ship, re-measure. If they're cosmetic and your volume is there, then you've earned the right to test. Either way you'll stop guessing, which is the whole point.
For the full diagnostic walkthrough, read our guide to running a landing page audit , and check the /sample-report to see what a finished one looks like.
Enough to collect the conversions a significance calculator says you need within about 4–6 weeks. As a principle, the lower your baseline conversion rate and the smaller the lift you want to detect, the more visitors per variant you need — often thousands of conversions per arm, not visitors. If you get only a handful of conversions a week, you don't have the volume yet; fix the page deterministically instead.
The four preconditions: a clear value proposition a stranger can state in 5 seconds, one dominant CTA, working and trustworthy analytics, and enough weekly conversions to reach significance. Beyond those, fix every deterministic issue first — missing trust signals, unanswered objections, an overlong form, a category-label headline. None of these need a test to confirm.
Usually one of two reasons: the variants were cosmetic (button color, a word swap) and too small to matter, or you didn't have enough conversions to detect a real difference, so the 'result' was noise. On low-traffic pages the second is the norm. Make bigger, fundamentals-level changes instead of testing tiny ones.
Rarely, until traffic grows. Small sites lack the conversion volume to reach statistical significance in a reasonable window, so tests run for months and get contaminated by changing traffic and seasonality. The higher-leverage move is a deterministic audit and shipping obvious fixes whose effect is large enough to see without a test.
An audit inspects your page against fixed criteria and tells you what's broken today — no traffic required. An A/B test compares two live versions and needs significant traffic to declare a winner. Audit first to fix deterministic problems; test later to settle close calls on a page that already works.
Run your page through the same diagnostic. The free preview takes about a minute — no signup.
Audit my page