A/B Testing Glossary: Statistics Terms, Explained.

Short, practitioner‑first definitions of the statistics terms you’ll see in A/B test results — sample size, significance, p‑values, SRM, peeking, and more — with practical notes for running tests.

Book a Demo
No credit card required 5.4KB script Privacy‑conscious, cookieless by default

Read, plan, and decide with confidence.

A practical glossary for people who run experiments — not a stats textbook.

Understand results

Decode p‑values, significance, intervals, and Bayesian confidence without the jargon.

Plan tests well

Size for your MDE, avoid underpowered tests, and keep guardrails honest.

Avoid pitfalls

Spot peeking and SRM early so you don’t ship false winners.

Decide pragmatically

Pair lift with uncertainty and business impact — not labels alone.

Sample size

Sample size is the number of visitors (or conversions) each variant needs before you can trust the result of an A/B test. It depends on baseline conversion rate, the smallest lift you care about (MDE), and the error rates you accept — not on a fixed “1,000 visitors” rule of thumb.

Underpowered tests look inconclusive or, worse, produce false positives that disappear after you ship. Oversized tests waste calendar time chasing lifts smaller than you would ever implement. Calculate sample size before launch using baseline rate, MDE, significance level (commonly α = 0.05), and power (commonly 80%).

In a Stellar test

Set your primary goal and traffic split up front. Estimate duration from page traffic and MDE, then avoid calling a winner until that plan is met. Read results using Bayesian confidence on the page.

Read more: Sample size for A/B testing
Plan yours with the sample size calculator →

Conversion rate

Conversion rate is conversions divided by visitors for a defined goal — for example, purchases ÷ visitors on a product page. In A/B testing it is the primary (or guardrail) metric you compare between control and variant.

Define the goal event and denominator before the test starts. Absolute conversion rate and relative lift answer different questions: a +10% relative lift on a 2% baseline is +0.2 percentage points absolute.

In a Stellar test

Pick one primary goal in the experiment setup and keep it fixed. Use secondary goals as guardrails, not for winner‑fishing after the fact.

Read more: CTR vs CR

Statistical significance

Statistical significance means the observed difference between variants is unlikely if there were truly no effect, under your pre‑chosen error threshold (often α = 0.05). It is not the same as “large enough to matter for the business.”

Significance answers a yes/no about surprise under the null; effect size and confidence intervals answer whether the lift is worth shipping. A tiny lift can be “significant” on huge traffic and still not be worth the implementation cost.

In a Stellar test

Read significance together with lift and your pre‑set MDE. Stellar reports results as Bayesian confidence (probability to beat control).

Read more: A/B testing significance

P‑value

A p‑value is the probability of seeing a difference at least as extreme as yours if the null hypothesis were true (usually “no difference”). A small p‑value means the data would be surprising under “no effect” — it does not mean the probability that B is better.

p‑values assume a fixed analysis plan. Repeated looks (peeking), optional stopping, and multiple undeclared metrics all distort them. Pair p‑values with intervals or Bayesian decision metrics when your tool provides them.

In a Stellar test

Prefer the product’s reported confidence / probability‑to‑beat‑control for decisions. Use classical p‑value explainers when reading external calculators or academic write‑ups.

Read more: P‑value explained

Confidence interval

A confidence interval is a range of plausible values for the true lift (or conversion rate difference) given your data and method. Prefer intervals over a lone “significant / not significant” label when deciding whether to ship.

If a 95% interval for lift runs from −0.2% to +1.8%, the data are compatible with both a small loss and a meaningful win — treat the test as inconclusive for a large ship decision. If the entire interval sits above your MDE, you have a stronger business case.

In a Stellar test

Compare the observed lift against your MDE. A confident result whose lift is smaller than the change you'd actually ship isn't a win worth rolling out.

MDE (minimum detectable effect)

MDE is the smallest relative or absolute lift your test is powered to detect reliably. Choose it from business value — the smallest change worth implementing — then size the test accordingly.

An unrealistically tiny MDE forces enormous samples. An oversized MDE means you will miss worthwhile wins. MDE, baseline rate, α, and power together determine sample size.

Practice tip

Before launch, write down the smallest lift you would actually ship (for example, +5% relative on trial starts). Use that as MDE when estimating duration from page traffic.

Statistical power

Statistical power is the probability of detecting a real effect of at least your MDE when that effect truly exists. 80% power is a common planning target; low power produces inconclusive tests and unstable “wins.”

Power rises with larger samples, larger true effects, and higher baseline rates (for binary metrics). Peeking does not legitimately increase power; it increases false positives.

Practice tip

If tests often end flat, raise MDE, extend runtime, focus on higher‑traffic pages, or improve the hypothesis — do not keep re‑checking the same underpowered test until it “goes green.”

Read more: Statistical power

Bayesian vs frequentist

Frequentist A/B testing summarizes evidence with p‑values and confidence intervals under a fixed sampling plan. Bayesian A/B testing updates a probability that a variant is better (or a posterior on lift) as data arrive.

Frequentist guarantees are easiest to interpret with pre‑committed sample sizes. Bayesian outputs (chance to beat control, expected loss) can be easier for product decisions — and often match how modern UIs are labeled — but still require honest priors and stopping rules.

In a Stellar test

Use Bayesian confidence / probability to beat control when explaining results. For deeper exploration, try the Bayesian A/B test calculator.

Read more: Bayesian vs frequentist

A/A test

An A/A test splits traffic between two identical experiences. You run it to validate assignment, tracking, and analysis — you should not see a meaningful winner if instrumentation is healthy.

A “significant” A/A result is a red flag for SRM, bot traffic, broken goals, or analysis bugs. A clean A/A that stays flat builds trust before you spend traffic on real variants.

Practice tip

Duplicate control as variant B (no visual changes), send traffic to the same goal, and verify counts and conversion rates look balanced.

False positive

A false positive (Type I error) is declaring a winning variant when there is no real difference. Your significance threshold (α) caps the long‑run false‑positive rate only if you follow the analysis plan it assumes.

Peeking, multiple undeclared metrics, multiple variants without correction, and underpowered tests all inflate false positives beyond the advertised 5%. Shipping false winners wastes engineering time and pollutes your experiment history.

Practice tip

Fix α and the primary metric before launch, avoid peeking, correct for multiple comparisons when needed, and validate big ships with a holdout or confirmation test.

Read more: Reducing false positives

SRM (sample ratio mismatch)

SRM means the observed traffic split between variants differs from the planned split (for example 55/45 when you set 50/50) by more than chance. SRM usually signals an assignment, redirect, caching, or tracking bug — not a real preference for one variant.

Analyze SRM with a chi‑squared or binomial test on visitor counts (not conversion rates). If SRM triggers, pause decision‑making and diagnose by browser, device, geo, and landing URL. Results under SRM are not trustworthy.

Practice tip

After the first day, compare visitor counts to the configured split and investigate any large mismatch before reading outcomes.

Peeking

Peeking is checking intermediate results and stopping (or changing) the test because the metrics look good or bad. Naive peeking inflates false positives because classical significance assumes a fixed sample size decided in advance.

Looking at dashboards for QA (traffic, errors) is fine. Using those looks to call a winner is not, unless you use a sequential / always‑valid method designed for optional stopping.

Practice tip

Monitor for broken variants and SRM early; ignore significance until the plan says you may decide.

Novelty effect

A novelty effect is a temporary change in behavior because the experience is new — not because it is better long term. Early lifts can fade once returning visitors adapt.

Mitigate by running through full weekly cycles, checking early‑period vs late‑period results, and preferring metrics that include return visits when the change is highly visible.

Practice tip

For loud UI changes, resist calling a winner in the first few days even if confidence looks high. Prefer a duration that covers at least one full business week.

Holdout

A holdout is a slice of traffic kept on the old experience after you think you have a winner, so you can confirm the lift still holds before a full rollout.

Holdouts catch winner’s curse, novelty fade, and implementation drift between the experiment variant and the production ship. They are especially useful after noisy or borderline tests.

Practice tip

After a win, keep a small control percentage for a confirmation period before applying the variant to 100%.

CUPED

CUPED (Controlled‑experiment Using Pre‑Experiment Data) is a variance‑reduction technique that uses data from before the experiment to reduce noise in the outcome metric, so you can detect the same effect with less traffic or detect smaller effects with the same traffic.

CUPED helps most when pre‑experiment behavior correlates strongly with the outcome (for example, prior clicks with CTR). It does not invent lift; it reduces variance. It is an analysis/engineering feature of some experimentation platforms.

Test your website without a developer.

Visual editor, targeting, and Bayesian confidence — set up in minutes.

Frequently Asked Questions