A/B Testing Rules: Run Tests That Mean Something

An A/B test is a controlled comparison, and it only produces a trustworthy answer when the rules are set before the traffic arrives. One control, one variant that differs in a defined way, a random split, a single primary metric, and a decision rule agreed in advance. Most disappointing tests fail on one of those conditions rather than on statistics. They change three things at once, measure whichever number moved, stop when someone gets nervous, and then declare a winner that will not reproduce. Treating experimentation as a discipline rather than a debate changes the quality of decisions almost immediately.

A test begins with a hypothesis specific enough to be wrong. More convincing proof on the checkout page will increase completed orders is testable; a cleaner design will perform better is not. Name the audience, the element, the change, and the metric. Tie the hypothesis to a plausible behavior, such as hesitation about shipping cost or doubt about product quality, and cite the observation that generated it. Both variants should be defensible versions of the page, because testing a serious option against an obviously weaker straw man produces uninformative wins and wastes the audience's trust.

Sample size is arithmetic, not opinion. You need the baseline conversion rate, the smallest effect the business cares about, the desired significance level, and the statistical power, usually eighty percent. A calculator gives the visitors per variant required to detect that effect. If a page converts at three percent and you want to detect a ten percent relative lift, the requirement is substantial, often tens of thousands of visitors per variant. When traffic cannot support the test, the honest options are testing bolder changes, running longer, or accepting a lower-powered decision framework with eyes open.

Plan duration by traffic and business cycles, not by the calendar convenience of a week. Cover at least one full week so weekday and weekend behavior both appear, and extend over pay cycles, promotions, or seasonal peaks if those affect buying. There is also a behavioral settling period: returning visitors may notice a change, and some audiences show novelty effects that fade. Decide the end date when the test starts and write it down. Checking results daily and stopping the moment a variant leads inflates false positives dramatically, because early differences are mostly noise.

Segment analysis is valuable and dangerous in equal measure. Reviewing results by device, source, and new versus returning visitors can reveal that a change helps mobile users and hurts desktop users, which is a genuinely useful finding. But slicing until something is significant guarantees a meaningless discovery, because each additional cut increases the chance of a random pattern passing the threshold. Define primary, secondary, and guardrail metrics before launch. If a segment result surprises you, treat it as a hypothesis for the next test, not as grounds to ship to that segment alone.

Statistical significance describes how likely a result this extreme would appear if the variants were truly identical; it does not measure the size or importance of the effect. A small lift on enormous traffic can be significant and irrelevant; a large lift on thin data can be promising and unproven. Report confidence intervals rather than a single threshold, and be honest about multiple comparisons when several metrics or segments are reviewed. Practical significance comes next: model the revenue impact, including margin and downstream quality, before deciding that a statistically real change is worth shipping.

Test validity depends on clean measurement. Confirm that every visitor is assigned to exactly one variant, that the assignment survives page reloads, and that both variants fire the same conversion tracking. Check for cross-contamination: a shared thank-you page, a global script that behaves differently, or a cache that serves the wrong version. Run the test on production rather than staging, since staging traffic rarely resembles real behavior. Before launching, submit a conversion in each variant and watch it land in analytics with the right parameters, because a broken test wastes weeks and often gets interpreted as a real result.

When the test ends, follow the decision rule that was written at the start. Ship the variant if it meets the criteria, keep the control if it does not, and document both outcomes in a shared log with the dates, traffic, effect size, and any anomalies. A losing test that rules out a plausible idea saves the cost of implementing it. Resist the temptation to rerun the same test with a slightly different wrinkle until it reaches significance; that is not experimentation, it is searching for a favorable sample. Trust across the organization is built by how losers are reported.

Wins are rarely permanent. Repeat validated changes once new traffic mixes arrive, and rerun important tests after redesigns, pricing changes, or audience shifts. Stack improvements deliberately: validate one change, ship it, and make it the new control for the next hypothesis. Keep a library of what has been tested, on which page, with what result, so new team members do not relitigate old questions. Over a year, a disciplined cadence of small, well-instrumented tests compounds into a clearer understanding of the audience than any single result could provide.

Some situations do not suit A/B testing. Very low traffic makes frequent tests futile; regulated or one-time pages leave little room for experimentation; and large strategic changes such as a rebrand are better evaluated with staged rollouts and before-and-after analysis. In those cases, use qualitative research, usability sessions, and cohort comparisons, and be explicit that the evidence is weaker. The purpose is not to run tests for their own sake; it is to make better decisions with the strongest available evidence, and sometimes that evidence comes from watching ten users struggle with a form.