
P Test Calculator: Get a P-Value for Your A/B Test

A p-value calculator takes your test data, or a test statistic such as a z or t score, and returns the probability of seeing a result at least this extreme if there were truly no difference between your variants. For an A/B test you need four numbers: visitors and conversions for the control, and visitors and conversions for the variant. This guide shows what to enter, how the math works by hand, how to read the answer, and where marketers go wrong.
TL;DR
- For conversion rates, use a two-proportion z-test calculator. For averages like order value, use a Welch's t-test. Enter unique users, not sessions.
- If p is at or below your alpha (usually 0.05), reject the null. If it is above, you have not shown a difference, which is not the same as proving there is none.
- A p-value says nothing about how big or valuable the effect is. Read the confidence interval next to it.
- Fix your sample size before the test starts. Checking the p-value daily and stopping when it dips under 0.05 inflates false positives.
- Our worked example below shows a 12% relative lift that still fails the 0.05 threshold.
Table of Contents
- What a p-value tells you about an A/B test
- Which test does your calculator need to run?
- What to enter, field by field
- A worked example: a 12% lift that is not significant
- Calculating a p-value from a t statistic
- How to read the result
- Mistakes that make a p-value lie
- Our perspective: the calculator is the last step, not the first
- Run the test, not just the calculation
- A quick checklist before you trust any p-value
- FAQ
- Sources
What a p-value tells you about an A/B test
A p-value is the probability of observing data as extreme as, or more extreme than, your sample, assuming the null hypothesis (no difference) is true. That definition is the standard one used by statistical calculators, and it is narrower than most people remember.
What it does not tell you:
- It is not the probability that your variant is better. Greenland and colleagues stress that a p-value is not the probability that a hypothesis is true, nor the probability that chance alone produced the result. It describes how well the data fit a model in which the test hypothesis holds.
- It is not the size of the effect. A tiny lift on a huge sample can give a very small p-value. A big lift on a small sample can give a large one.
- A high p-value is not proof of no effect. The same paper points out that P > 0.05 only means the null is one of many hypotheses compatible with the data.
The logic is proof by contradiction. As Cyril Pernet's tutorial describes it, you pose a null model and test whether the collected data conform to it. A small p-value says the data fit that model poorly. That is all it says. For a longer walkthrough aimed at marketers, see our p-value explainer.
Which test does your calculator need to run?
Searching "p test calculator" returns a mix of tools, and they are not interchangeable. Some convert a test statistic you already have into a p-value. Others take raw data and run the test themselves. A few, like series convergence tools, have nothing to do with statistics. Pick by what you are holding.
| You have | You need | Typical inputs |
|---|---|---|
| A z score, t score, F or chi-square value | A test-statistic-to-p-value calculator | The statistic, degrees of freedom (for t, F, chi-square), one or two tails |
| Visitors and conversions per variant | A two-proportion z-test | Users and conversions for each group, alpha, tails |
| Mean, standard deviation and count per variant (revenue per user, order value) | A Welch's t-test | Mean, SD, n for each group |
| Raw rows of data | A t-test or other test on the dataset | Both columns of values |
The two A/B cases are the ones that matter most for conversion work. One A/B significance calculator processes two-proportion z-tests for conversion rates and Welch's t-tests for averages such as order value. Another post hoc calculator switches automatically to Welch's t-test when sample sizes differ a lot or when a group under 30 has unequal sizes.
On sample size, that same source says a normal approximation is generally safe above 30 per group unless the data are extremely skewed, and that Student's t-test is mostly recommended below 30. In practice, most web tests have thousands of users per arm, so the z-test for proportions is your default.
What to enter, field by field
Good inputs matter more than a fancy calculator. Here is what each field should hold.
- Visitors per variant. Use unique users, not sessions. Counting sessions lets one person appear several times, which breaks the independence assumption behind the math. Common mistakes include using session counts instead of unique user counts.
- Conversions per variant. Count each user once, against the one goal you decided on before the test. If you are unsure which action to measure, our guide to choosing conversion actions helps.
- Alpha. The threshold for calling a result significant. 0.05 is the convention, with 0.01 and 0.10 also common.
- One-tailed or two-tailed. Use two-tailed unless you committed in advance to only caring about one direction. A two-sided test catches a variant that is worse as well as better, which is usually what you want to know.
- Number of comparisons. If you test several variants against one control, the calculator should adjust for multiple comparisons. Otherwise the chance that at least one variant looks like a winner by luck climbs quickly.
If you are comparing the conversions from two ad or landing page variants, the broader conversion rate optimization tips give useful context on which changes are worth testing in the first place.
A worked example: a 12% lift that is not significant
Say your control page gets 10,000 unique visitors and 500 conversions (5.0%). The variant also gets 10,000 visitors and 560 conversions (5.6%). That is 0.6 percentage points, or a 12% relative lift. Is it real?
Step 1: state the hypotheses. Null: both pages convert at the same rate. Alternative: the rates differ (two-sided). Alpha: 0.05.
Step 2: pool the rate. Under the null, both groups share one true rate. Pooled rate = (500 + 560) / 20,000 = 0.053.
Step 3: compute the standard error. SE = square root of [0.053 × 0.947 × (1/10,000 + 1/10,000)] ≈ 0.00317.
Step 4: compute the z statistic. z = (0.056 − 0.050) / 0.00317 ≈ 1.89.
Step 5: convert z to a p-value. The area in one tail beyond 1.89 on the standard normal curve is about 0.029. Because the test is two-sided, double it: p ≈ 0.058.
The doubling step is how the Penn State course material describes two-sided tests: find the tail area, then multiply by two.
Step 6: compare to alpha. 0.058 is above 0.05, so you do not reject the null. The data are compatible with no difference, and also with a real lift.
The 95% confidence interval for the difference runs from roughly −0.02 to +1.22 percentage points. That range is more informative than the single p-value. It says the variant could be flat, or it could be worth more than a point of conversion. The honest conclusion is "not enough evidence yet," not "the variant failed."

Note how fragile the verdict is. A few dozen more conversions in the variant would push p below 0.05. That is exactly why people are tempted to keep checking, which brings us to the biggest trap.
Calculating a p-value from a t statistic
If your tool gave you a t score and you want the p-value, the process has two parts. First identify the degrees of freedom. Then find the area under the t-distribution curve beyond your observed statistic. As the Penn State material puts it, for a two-sided test you calculate that tail area and multiply by two. A t-to-p walkthrough describes the same steps: find the degrees of freedom, find the upper tail probability, double it.
A converter calculator does this for you. A statistic-to-p-value tool uses the standard normal curve for z-scores, the t-distribution for t-statistics and the chi-square distribution for chi-square statistics, and asks for degrees of freedom for the last two.
The full recipe, per the same Penn State page, is: define the null and alternative, compute the test statistic, find the probability of an outcome that extreme under the null, and compare it with alpha. For a deeper take specific to experiments, see our guide on how to calculate a p-value in A/B testing.
How to read the result
The decision rule is short. Compare the p-value with the alpha you chose before the test. If p ≤ alpha, reject the null. If p > alpha, do not reject it, per the Penn State guidance.
Three habits make the reading better.
Look at the interval, not just the verdict. One calculator maker notes that confidence intervals are often more useful than p-values because they show the range of effect sizes consistent with the data, so you can judge whether the effect is big enough to matter. Greenland et al. likewise urge reading intervals alongside p-values.
Separate statistical from practical significance. A p of 0.001 on a 0.05% lift may be real and still not worth shipping if the change costs engineering time or hurts something else. Ask what the lift is worth in revenue.
Do not lean on a single number. Andrade argues that decisions should not rest on p-values alone, since they do not estimate the probability that a hypothesis is true or that chance is solely responsible for the results. Our own piece on interpreting statistical significance covers the decision side in more detail.
One more point: a good significance calculator reports numbers and does not crown a winner. Some are built that way on purpose, reporting p-values, confidence intervals, relative lift and planned-effect power, and leaving the call to you.

Mistakes that make a p-value lie
Peeking and stopping early
This is the most common one. You open the dashboard every morning and stop the moment p drops under 0.05. The calculator documentation is blunt: optional stopping inflates false-positive rates far beyond the nominal alpha, and results are valid only at a sample size fixed in advance. A second maker of calculators warns that significance calculators suit fixed-horizon experiments; otherwise the models become invalid.
The fix: decide the sample size and duration up front, run the full period, then calculate once. If you must monitor continuously, use a method designed for sequential looks rather than a fixed-horizon calculator. Our article on avoiding false positives goes deeper.
Skipping the power calculation
A test that is too small can only detect huge effects, so a non-significant result tells you little. Work out the sample you need for the smallest lift you care about before launching. We cover the how in calculating power for A/B tests and the concept in statistical power for marketers.
Testing many variants without correction
Four variants against one control means several chances for a fluke. Use a calculator that accounts for the number of comparisons, or tighten your alpha.
Breaking independence
Sessions, repeat visitors and the same person seeing both variants all violate the assumption that observations are independent. Randomize at the user level and count users.
Treating a non-significant result as a verdict
P above 0.05 means "unproven," not "no effect." Check the interval and your power before you bin the idea. For a wider checklist, see 7 common A/B testing mistakes.
Our perspective: the calculator is the last step, not the first
We build an A/B testing platform, and the pattern we see is consistent: teams reach for a p-value calculator at the end of a test, after the decisions that determine whether the number means anything are already made. The goal, the sample size and the stopping rule are what make the p-value trustworthy.
So we push the same few habits. Pick one primary goal before launch. Estimate the sample you need for the smallest lift worth shipping. Let the test run its planned length. Then read the p-value together with the interval, and ask what the effect is worth.
We also think speed belongs in the statistics conversation. A heavy testing script can slow pages, change behavior and quietly contaminate the very conversion rate you are measuring. That is why we built Stellar around a lightweight script, with a no-code visual editor so marketers can launch experiments without waiting on developers, and real-time analytics so the numbers are there when the planned sample is reached. Real-time does not mean you should stop early. It means you can check that tracking works from day one, then wait.
— Juan
Run the test, not just the calculation
A calculator tells you what happened in data you already have. If you want to collect that data cleanly, Stellar gives you a no-code visual editor, goal tracking and real-time analytics, with direct integrations for WordPress, Shopify, Webflow, Wix, Squarespace, Framer and Bubble. Plans are priced by monthly tracked users: Sandbox is $0 per month, Plus is $29 per month, Pro is $99 per month and Scale is $399 per month. Enterprise has no published price. The free Sandbox plan is aimed at sites with under 25,000 monthly tracked users, so you can try a first experiment before paying anything.
A quick checklist before you trust any p-value
- One primary goal, chosen before launch.
- Sample size and duration fixed in advance, based on the smallest lift you care about.
- Unique users counted, not sessions.
- Two-tailed test unless you committed to one direction beforehand.
- Adjustment for multiple variants.
- No early stopping on a lucky dip.
- p-value read alongside the confidence interval and the business value of the lift.
If you want a second reference on the calculator side, our A/B test calculator guide covers a related workflow, though it is written in German.
FAQ
How do you calculate a p-value from a test statistic?
Find the degrees of freedom if the distribution needs them, then find the area in the tail beyond your statistic. For a two-sided test, double that tail area. A calculator does this instantly once you enter the z, t, F or chi-square value.
What alpha should I use for an A/B test?
0.05 is the usual default, with 0.01 for stricter decisions and 0.10 for lower-stakes ones. Choose it before the test and keep it fixed. If you compare several variants, tighten it or use a calculator that corrects for multiple comparisons.
Is a p-value of 0.05 or lower proof that my variant won?
No. It means the data would be fairly unusual if there were no difference, but it does not give the probability that the variant is better, and it says nothing about effect size. Check the confidence interval and the business value of the lift before you ship.
Why does my calculator say not significant when the variant clearly converts higher?
A visible lift can still be within the range of random variation, especially with small samples. In our example, a 12% relative lift on 10,000 users per arm gives p of about 0.058. Either collect the sample you planned or accept that the evidence is not strong enough yet.
Sources
- A/B Test Significance Calculator, p-value and confidence intervals
- Statistical tests, P values, confidence intervals, and power (Greenland et al., 2016)
- Null hypothesis significance testing: a short tutorial (Pernet)
- Hypothesis testing, the p-value approach (Penn State STAT ONLINE)
- P-value calculator from z, t or chi-square (StatsLearn)
- Find the p value from a t value (StatKat)
- P-value and statistical significance calculator (Georgiev)
- The P Value and Statistical Significance (Andrade, 2019)