Sample size
Sample size is the number of visitors (or conversions) each variant needs before you can trust the result of an A/B test. It depends on baseline conversion rate, the smallest lift you care about (MDE), and the error rates you accept — not on a fixed “1,000 visitors” rule of thumb.
Underpowered tests look inconclusive or, worse, produce false positives that disappear after you ship. Oversized tests waste calendar time chasing lifts smaller than you would ever implement. Calculate sample size before launch using baseline rate, MDE, significance level (commonly α = 0.05), and power (commonly 80%).
In a Stellar test
Set your primary goal and traffic split up front. Estimate duration from page traffic and MDE, then avoid calling a winner until that plan is met. Read results using Bayesian confidence on the page.
Read more: Sample size for A/B testing
Plan yours with the sample size calculator →Conversion rate
Conversion rate is conversions divided by visitors for a defined goal — for example, purchases ÷ visitors on a product page. In A/B testing it is the primary (or guardrail) metric you compare between control and variant.
Define the goal event and denominator before the test starts. Absolute conversion rate and relative lift answer different questions: a +10% relative lift on a 2% baseline is +0.2 percentage points absolute.
In a Stellar test
Pick one primary goal in the experiment setup and keep it fixed. Use secondary goals as guardrails, not for winner‑fishing after the fact.
Read more: CTR vs CRStatistical significance
Statistical significance means the observed difference between variants is unlikely if there were truly no effect, under your pre‑chosen error threshold (often α = 0.05). It is not the same as “large enough to matter for the business.”
Significance answers a yes/no about surprise under the null; effect size and confidence intervals answer whether the lift is worth shipping. A tiny lift can be “significant” on huge traffic and still not be worth the implementation cost.
In a Stellar test
Read significance together with lift and your pre‑set MDE. Stellar reports results as Bayesian confidence (probability to beat control).
Read more: A/B testing significanceP‑value
A p‑value is the probability of seeing a difference at least as extreme as yours if the null hypothesis were true (usually “no difference”). A small p‑value means the data would be surprising under “no effect” — it does not mean the probability that B is better.
p‑values assume a fixed analysis plan. Repeated looks (peeking), optional stopping, and multiple undeclared metrics all distort them. Pair p‑values with intervals or Bayesian decision metrics when your tool provides them.
In a Stellar test
Prefer the product’s reported confidence / probability‑to‑beat‑control for decisions. Use classical p‑value explainers when reading external calculators or academic write‑ups.
Read more: P‑value explainedConfidence interval
A confidence interval is a range of plausible values for the true lift (or conversion rate difference) given your data and method. Prefer intervals over a lone “significant / not significant” label when deciding whether to ship.
If a 95% interval for lift runs from −0.2% to +1.8%, the data are compatible with both a small loss and a meaningful win — treat the test as inconclusive for a large ship decision. If the entire interval sits above your MDE, you have a stronger business case.
In a Stellar test
Compare the observed lift against your MDE. A confident result whose lift is smaller than the change you'd actually ship isn't a win worth rolling out.
MDE (minimum detectable effect)
MDE is the smallest relative or absolute lift your test is powered to detect reliably. Choose it from business value — the smallest change worth implementing — then size the test accordingly.
An unrealistically tiny MDE forces enormous samples. An oversized MDE means you will miss worthwhile wins. MDE, baseline rate, α, and power together determine sample size.
Practice tip
Before launch, write down the smallest lift you would actually ship (for example, +5% relative on trial starts). Use that as MDE when estimating duration from page traffic.
Statistical power
Statistical power is the probability of detecting a real effect of at least your MDE when that effect truly exists. 80% power is a common planning target; low power produces inconclusive tests and unstable “wins.”
Power rises with larger samples, larger true effects, and higher baseline rates (for binary metrics). Peeking does not legitimately increase power; it increases false positives.
Practice tip
If tests often end flat, raise MDE, extend runtime, focus on higher‑traffic pages, or improve the hypothesis — do not keep re‑checking the same underpowered test until it “goes green.”
Read more: Statistical powerBayesian vs frequentist
Frequentist A/B testing summarizes evidence with p‑values and confidence intervals under a fixed sampling plan. Bayesian A/B testing updates a probability that a variant is better (or a posterior on lift) as data arrive.
Frequentist guarantees are easiest to interpret with pre‑committed sample sizes. Bayesian outputs (chance to beat control, expected loss) can be easier for product decisions — and often match how modern UIs are labeled — but still require honest priors and stopping rules.
In a Stellar test
Use Bayesian confidence / probability to beat control when explaining results. For deeper exploration, try the Bayesian A/B test calculator.
Read more: Bayesian vs frequentistA/A test
An A/A test splits traffic between two identical experiences. You run it to validate assignment, tracking, and analysis — you should not see a meaningful winner if instrumentation is healthy.
A “significant” A/A result is a red flag for SRM, bot traffic, broken goals, or analysis bugs. A clean A/A that stays flat builds trust before you spend traffic on real variants.
Practice tip
Duplicate control as variant B (no visual changes), send traffic to the same goal, and verify counts and conversion rates look balanced.
False positive
A false positive (Type I error) is declaring a winning variant when there is no real difference. Your significance threshold (α) caps the long‑run false‑positive rate only if you follow the analysis plan it assumes.
Peeking, multiple undeclared metrics, multiple variants without correction, and underpowered tests all inflate false positives beyond the advertised 5%. Shipping false winners wastes engineering time and pollutes your experiment history.
Practice tip
Fix α and the primary metric before launch, avoid peeking, correct for multiple comparisons when needed, and validate big ships with a holdout or confirmation test.
Read more: Reducing false positivesSRM (sample ratio mismatch)
SRM means the observed traffic split between variants differs from the planned split (for example 55/45 when you set 50/50) by more than chance. SRM usually signals an assignment, redirect, caching, or tracking bug — not a real preference for one variant.
Analyze SRM with a chi‑squared or binomial test on visitor counts (not conversion rates). If SRM triggers, pause decision‑making and diagnose by browser, device, geo, and landing URL. Results under SRM are not trustworthy.
Practice tip
After the first day, compare visitor counts to the configured split and investigate any large mismatch before reading outcomes.
Peeking
Peeking is checking intermediate results and stopping (or changing) the test because the metrics look good or bad. Naive peeking inflates false positives because classical significance assumes a fixed sample size decided in advance.
Looking at dashboards for QA (traffic, errors) is fine. Using those looks to call a winner is not, unless you use a sequential / always‑valid method designed for optional stopping.
Practice tip
Monitor for broken variants and SRM early; ignore significance until the plan says you may decide.
Novelty effect
A novelty effect is a temporary change in behavior because the experience is new — not because it is better long term. Early lifts can fade once returning visitors adapt.
Mitigate by running through full weekly cycles, checking early‑period vs late‑period results, and preferring metrics that include return visits when the change is highly visible.
Practice tip
For loud UI changes, resist calling a winner in the first few days even if confidence looks high. Prefer a duration that covers at least one full business week.
Holdout
A holdout is a slice of traffic kept on the old experience after you think you have a winner, so you can confirm the lift still holds before a full rollout.
Holdouts catch winner’s curse, novelty fade, and implementation drift between the experiment variant and the production ship. They are especially useful after noisy or borderline tests.
Practice tip
After a win, keep a small control percentage for a confirmation period before applying the variant to 100%.
CUPED
CUPED (Controlled‑experiment Using Pre‑Experiment Data) is a variance‑reduction technique that uses data from before the experiment to reduce noise in the outcome metric, so you can detect the same effect with less traffic or detect smaller effects with the same traffic.
CUPED helps most when pre‑experiment behavior correlates strongly with the outcome (for example, prior clicks with CTR). It does not invent lift; it reduces variance. It is an analysis/engineering feature of some experimentation platforms.