Calculate Statistical Significance: 7 Inputs Marketers Need

Marketer entering experiment values into calculator

A result is statistically significant when its p-value is at or below your chosen alpha, commonly around 0.05. To check this quickly, run a two-proportion z test on conversion counts or a t-test on means, then look past the p-value alone to the confidence interval and effect size before deciding anything matters.

TL;DR:

Use a z test for conversion counts, a t test for means, paired tests for repeated users, or ANOVA across three or more groups.

Choose whether to test one direction or either direction before reviewing results; alpha 0.05 is standard, while 0.01 suits decisions with high stakes.

A p value does not show the probability that the null is true; pair it with confidence intervals and effect size to assess practical impact.

Plan power and sample size before launch around the minimum effect worth detecting; an underpowered nonsignificant result does not establish that no effect exists.

Correct for multiple comparisons and repeated result checks, since testing many metrics or stopping when significance appears raises false positive risk.

What to enter into a statistical significance calculator
Choose the right test and understand the basic formulas
How to interpret p-values, confidence intervals, effect size, and power
A concrete A/B example: computing significance from event counts
Common pitfalls and reporting best practices
Practical tips for marketers running experiments
Author perspective: statistical significance as a decision aid
Try Stellar to run and measure A/B tests
FAQ
Sources

What to enter into a statistical significance calculator

Before you touch a calculator, figure out what kind of metric you have. A conversion rate, click-through rate, or any yes/no outcome is binary data. Average order value, time on page, or revenue per visitor is continuous data, and it needs a different test and different inputs.

Once you know the metric type, gathering the right numbers makes the rest of the process almost mechanical.

Pick the metric type: binary events (conversions, clicks) or continuous values (averages, revenue).
Collect raw counts or summary stats: for binary data, grab conversions and total visitors per group; for continuous data, grab the mean, standard deviation, and sample size per group.
Choose sidedness: a two-sided test checks for any difference; a one-sided test checks for a difference in one direction only, and you should decide this before you see results.
Set alpha: 0.05 is standard, though 0.01 is common in high-stakes decisions.
Account for uneven allocation: note your traffic split (50/50, 90/10) since it affects the standard error.
Confirm independence or pairing: unpaired groups (new visitors in each arm) need a standard two-sample test; the same users measured twice need a paired test.
Run the calculator and read every output: p-value, confidence interval, effect size, and power, not just a "significant" or "not significant" label.

Choose the right test and understand the basic formulas

The test you pick depends entirely on your data type and design. Calculators handle the arithmetic, but choosing the wrong test produces a confident, wrong answer.

Two-proportion z test: for independent conversion-rate comparisons, it pools the two groups' variance under the null hypothesis and expresses the difference between rates in standard-error units.
T-tests: one-sample, two-sample, or paired, used for means when the population variance is unknown and estimated from the sample.
Chi-square or exact tests: for categorical contingency tables, especially with small cell counts where normal approximations break down.
ANOVA: for comparing means across three or more groups at once, rather than running repeated pairwise t-tests.
Nonparametric or exact binomial methods: when sample sizes are small or data is heavily skewed, these avoid relying on a normal approximation that may not hold.

One figure worth remembering: according to statistical test selection guidance, the z statistic for a proportion test expresses the difference from a hypothesized rate in standard-error units, and one-sided tests are only valid when the direction is planned in advance, not chosen after seeing the data.

Picking the test is really about answering three questions: Is my outcome binary or continuous? Are my groups independent or paired? How many groups am I comparing? Once those are settled, the formula the calculator uses is determined, not a matter of preference.

How to interpret p-values, confidence intervals, effect size, and power

A p-value tells you how compatible your data is with the null hypothesis, nothing more. According to the American Statistical Association's guidance on p-values, a p-value is not the probability that the null hypothesis is true, and a small p-value does not guarantee a large or practically important effect.

Confidence intervals do more work than a single p-value. Penn State's course materials on hypothesis testing note that a 95% confidence interval excluding the null value typically aligns with rejecting that null at alpha = 0.05, and the interval shows the full range of effect sizes compatible with your data, which is far more useful for judging practical impact than a pass or fail label.

Power matters before you launch, not after. It's a planning number, the probability of detecting a real effect of a given size, and reading it after the fact as proof your result is solid misunderstands what it measures.

Report the p-value, confidence interval, and effect size together, never the p-value alone.
Treat power as something you calculate before the test, using your baseline rate and minimum effect worth detecting.
Adjust for multiple comparisons when you're testing several metrics or variants at once.

Pro Tip: Read the confidence interval before the p-value. It tells you the range of plausible uplifts, which matters more for a shipping decision than a single threshold.

A concrete A/B example: computing significance from event counts

Here's the sequence a calculator runs:

Pool the proportion: combine both groups' conversions over total visitors to get the pooled rate under the null hypothesis of no difference.
Calculate the standard error: using the pooled proportion and both sample sizes.
Compute the z statistic: the difference between the two observed rates divided by that standard error.
Convert the z statistic to a p-value: using the standard normal distribution.
Build the confidence interval: around the observed difference in rates, usually at 95%.
Calculate percent uplift: the variant's 12.5% versus control's 10% is a relative uplift worth stating alongside the raw percentage-point difference.

The outcome to watch for: according to Penn State's lesson on test statistics, a result is statistically significant when the p-value falls at or below alpha, commonly around 0.05, 0.01, or 0.10 depending on the stakes involved.

If the p-value comes in above your alpha, check whether your sample size was large enough to detect the uplift you're hoping for before concluding there's no effect. An underpowered test and a true null look identical from the p-value alone.

A concrete A/B example: computing significance from event counts — overview diagram

Common pitfalls and reporting best practices

Most invalid conclusions come from a handful of repeatable mistakes, not from bad math.

Don't switch to a one-sided test after seeing the data. Sidedness and alpha get decided before you run the experiment, not after.
Don't peek and stop early without correction. Checking results repeatedly and stopping as soon as you see significance inflates your false positive rate well above your stated alpha.
Report effect size and confidence interval alongside the p-value, every time, not just when the result looks good.
Adjust for multiple comparisons when testing several metrics, segments, or variants in the same experiment.

Pro Tip: Decide your stopping rule and sample size before launch, and write them down. It's the single easiest way to avoid fooling yourself later.

The American Statistical Association frames this clearly: thresholds exist to aid judgment, not replace it, and decisions should never rest on whether a p-value crosses 0.05 in isolation.

Practical tips for marketers running experiments

Collect raw events and visitor counts, or means and standard deviations, exactly as your calculator expects, and log them with the test's start and end dates. Decide your minimum effect worth shipping before you look at results, then weigh that figure against the confidence interval, not just the point estimate, using conversion rate optimization tips to guide your practical choices. Avoid checking results on inconsistent windows or reporting only the metrics that moved. For a deeper playbook on reading these outputs, see our guide to interpreting statistical significance in A/B results.

Author perspective: statistical significance as a decision aid

A p-value is one input, not a verdict. The sharpest growth teams pair it with replication and a minimum effect worth shipping, instead of chasing a threshold.

— Juan

Try Stellar to run and measure A/B tests

Running the test is half the job; collecting clean counts and means to feed a calculator is the other half. Our no-code visual editor tracks conversions and goals in real time, with a free Sandbox plan for teams under 25,000 monthly tracked users.

Gostellar

Once you've planned your sample size and alpha, launching inside Stellar takes minutes, and you can compare plans or start free whenever you're ready to measure real traffic instead of a spreadsheet.

FAQ

How do you calculate statistical significance?

Compare your test's p-value to your chosen alpha, typically 0.05. Run a two-proportion z test for conversion counts or a t-test for means, then check the confidence interval and effect size alongside the p-value before deciding the result is meaningful.

How do you calculate the 0.05 level of significance?

Set alpha at 0.05 before running your test, then compute the test statistic (z, t, or chi-square) from your sample data. If the resulting p-value is at or below 0.05, the result is considered statistically significant at that level.

What is a statistical significance calculator?

A statistical significance calculator takes raw inputs like conversion counts, visitor totals, or means and standard deviations, then computes a test statistic and p-value automatically. It typically also returns a confidence interval and effect size, which matter more than the p-value alone when judging practical impact.

What sample size do I need before running a test?

Sample size depends on your baseline rate, the minimum effect you want to detect, and your chosen alpha and power, usually calculated with tools like statsmodels' power functions for two-proportion tests. Our power and sample-size guide walks through the specific inputs needed.

What should I report alongside a p-value?

Report the confidence interval, effect size, and sample sizes for each group, not the p-value in isolation. The Penn State comparison of confidence intervals and hypothesis tests notes that intervals show the full range of compatible effect sizes, which is more useful for judging practical significance.

Try Stellar A/B Testing for Free!