P Value Calculator

What a p value answers

A p value answers one narrow question: if the null hypothesis were true, how often would a test produce a result at least as far from the null as the one in front of me? That is close to the wording used by the NIST engineering statistics handbook, and the sentence is worth reading twice, because it is about the data, not about the hypothesis.

Take a z score of 2.1. On a normal curve, 3.57 percent of the area sits beyond 2.1 in both directions, so the two-tailed p value is 0.0357. If the null were true, a result this far out would show up about 3.6 times in 100 tries. That is the whole meaning of the number. It is not the chance that the null is true, and it is not the chance the result happened by luck.

Small p values mean the data sit far from what the null predicts. Large p values mean the data are unremarkable under the null, which is not the same as proving the null. A test can only fail to find evidence, and a small sample fails to find evidence all the time, even when a real difference exists.

The threshold does the deciding, not the p value by itself. NIST notes that 0.10, 0.05 and 0.01 are common choices and that the choice is somewhat arbitrary, which is why a result at 0.049 and one at 0.051 describe almost the same data while landing on opposite sides of a line.

The test statistic comes first

Every p value is built from a test statistic and a reference distribution. The statistic measures how far the data sit from the null in units of noise, and the distribution turns that distance into an area. A handful of shapes cover most of what people need.

The denominator is always a standard error, which is the spread of the estimate divided by the square root of the sample size. That square root is the reason sample size dominates the discussion later on: to halve a standard error you need four times the data, not twice.

Degrees of freedom control how heavy the tails of the t curve are. With 5 degrees of freedom the t curve has fatter tails than the normal curve, so the same distance produces a larger p value. By 30 degrees of freedom the two curves are close, and many textbooks switch to the normal curve there, though software keeps using t.

Worked example: a z score of 2.1

A factory weighs filled bags and compares a sample mean against a target. The process spread is known from years of records, so the statistic is a z score. The sample mean comes out 0.2 pounds above target, the standard error is 0.0952 pounds, and the arithmetic is one division.

• z = 0.2 / 0.0952 = 2.101
• Area beyond 2.101 in one direction: 0.0179
• Two-tailed p value: 0.0179 x 2 = 0.0357
• Call at the 0.05 level: reject the null
• Call at the 0.01 level: do not reject

The same distance means different things at different thresholds, which is why a p value should be reported as a number instead of a verdict. A reader who wants a stricter test can apply it without rerunning anything.

Notice how fast the area falls. Moving from 1.5 to 2.5 cuts the two-tailed p from 0.1336 to 0.0124, a factor of about 10.8, while the z score itself rises by two thirds. Tail areas shrink faster than the distance grows, which is why p values feel dramatic near the threshold and why the gap between 0.04 and 0.001 is wider than the numbers suggest.

Worked example: 25 bags of coffee

A roaster claims each bag holds 5.0 ounces. A buyer weighs 25 bags and finds a mean of 5.2 ounces with a standard deviation of 0.6 ounces. The spread now comes from the sample, so the test uses t.

• Standard error: 0.6 / 5 = 0.12
• t = (5.2 - 5.0) / 0.12 = 1.667
• Degrees of freedom: 25 - 1 = 24
• Two-tailed p value: 0.109
• Call at 0.05: not significant

The sample mean is high by 0.2 ounces, yet the p value says a gap this size shows up about 11 times in 100 when the fill is honest. Twenty five bags carry too much noise to separate a 4 percent shortfall from ordinary variation.

Now keep the gap and the spread and change only the sample size. A hundred bags make the standard error 0.06, the t score doubles to 3.333, and the p value falls to 0.0012. Nothing about the coffee changed. Only the amount of evidence did, which is the most useful thing to understand about p values.

Two warnings about that table. Ten bags is a thin sample for estimating a standard deviation, so its p value deserves little weight. And a significant result on 100 bags still says nothing about whether 0.2 ounces matters to a shopper who cannot taste the difference, which is the effect size question covered below.

One tail or two tails

A one-tailed p value counts only the far end of the curve in the direction you predicted. A two-tailed p value counts both ends, so it is exactly twice the one-tailed figure on a symmetric curve. A z of 2.1 therefore gives 0.0179 one-tailed and 0.0357 two-tailed.

Pick the tail count before you look at the data. Two tails answer the question most people have, which is whether a difference exists in either direction. One tail is defensible when a result in the other direction would be ignored or would count against the same claim, such as testing whether a new seal leaks more than the old one.

What is not defensible is switching. A result of 0.07 two-tailed becomes 0.035 if you decide after the fact that you only care about one direction. That move turns a null result into a headline without new data, and anyone who spots it will discount the report.

The gap between the z column and the t columns is the price of not knowing the population spread. With 10 degrees of freedom, a t of 2.228 is needed to reach the same 0.05 as a z of 1.960, roughly 14 percent further from the null. That extra distance exists because the sample standard deviation is itself an estimate, and it can come out too small by chance.

What a p value cannot tell you

The American Statistical Association statement on p values from 2016 lists six principles, and three of them address the common misreadings head on. A p value does not measure the probability that the hypothesis is true. It does not measure the probability that the data came from random chance alone. And it does not measure the size or the importance of an effect.

Each error is easy to make out loud. Saying that p = 0.03 means a 97 percent chance the treatment works flips the conditional: the p value is computed assuming the null, not the other way around.

Effect size is a separate question. A pill that drops 0.1 pounds per person becomes significant on a sample of 200,000 people, and nobody would call that result meaningful. Reporting the gap in its own units, next to a confidence interval, gives readers what the p value leaves out.

Multiple testing is the other trap. With an alpha of 0.05, each clean test still fires 5 percent of the time. Run 20 independent tests on data with no real effect and the chance of at least one false positive is 1 - 0.95 to the 20th power, which is 0.6415, or about 64 percent. That arithmetic is why a single striking result from a wide search deserves a repeat before it deserves belief.

A p value of 0.049 and one of 0.051 carry almost the same information, so a rule that treats them as different outcomes is a rule about paperwork. NIST puts it plainly: decide the threshold before the test, then report the number itself.

P values, intervals and sample size

A two-tailed p value and a confidence interval are two views of one calculation. If the 95 percent interval for a difference excludes zero, the two-tailed p value is below 0.05. Slide the interval until it touches zero and the p value reaches 0.05, not by coincidence but by construction.

The coffee example shows the link with numbers. The gap is 0.2 ounces and the standard error is 0.12, so the 95 percent interval is 0.2 plus or minus 2.064 x 0.12. That works out to 0.2 plus or minus 0.248, which spans -0.048 to 0.448. The interval contains zero, and the matching p value is 0.109, above 0.05.

On 100 bags the interval is 0.2 plus or minus 1.984 x 0.06, or 0.081 to 0.319. Zero is outside, and the p value is 0.0012. The interval also shows why the result is thin: the lower bound of 0.081 ounces is a gap a shopper would never notice, so the finding is real in the statistical sense and small in the practical one.

Sample size planning runs the same math backwards. Decide the smallest gap worth detecting, guess the spread, and choose the sample that puts the standard error small enough for that gap to reach significance. A pilot of 25 bags cannot detect a 0.2 ounce shortfall at the 0.05 level when the spread is 0.6, and no amount of analysis fixes that after the fact. A review of the ASA principles in clinical research makes the same point for trials: an underpowered study that reports a large p value has not shown that the effect is absent.