P-value

A p-value is calculated from a test statistic and a reference distribution chosen for a statistical test. It is the probability, assuming the null hypothesis and the test's other assumptions, of obtaining a statistic at least as extreme as the observed statistic. What counts as “extreme” depends on the test and the direction or directions specified by the alternative hypothesis. A small p-value means the observed data would be relatively unusual under that particular model; it does not by itself say why the data are unusual.

Suppose a fair coin is tossed 10 times and a one-sided test was specified in advance to look for an excess of heads. If 8 heads are observed, the p-value is the probability of 8 or more heads under the fair-coin model: [C(10,8) + C(10,9) + C(10,10)] / 2^10 = 56/1024, or about 0.0547. For this test, the result is not statistically significant at a preselected 0.05 level. The p-value does not mean that there is a 5.47% chance the coin is fair; it is a probability about possible outcomes under the assumed fair-coin model and specified test.

Researchers often compare a p-value with a significance level, commonly written α, chosen for a test. If the procedure and assumptions are appropriate, α sets the long-run probability of rejecting the null hypothesis when it is true. In a test using α = 0.05, a p-value below 0.05 is conventionally called statistically significant. The threshold is a decision convention, not a boundary between true and false explanations; results just above and below it need not be meaningfully different. A p-value and α are related but distinct: α is selected before assessing the result, while the p-value is computed from the observed data.

A p-value is not the probability that the null hypothesis is true, the probability that a result occurred “by chance alone,” or the probability that a finding will replicate. It does not measure the size, importance, or practical value of an effect. A low value can arise from a small effect measured precisely, while a larger value may reflect limited data or imprecision; interpretation requires estimates and context. The American Statistical Association emphasizes that scientific conclusions should not rest solely on whether a p-value crosses a fixed threshold.

A p-value depends on the model, test statistic, test direction, data collection, and assumptions such as independence or the form of a sampling distribution. Choosing analyses after seeing the data, selectively reporting tests, or conducting many tests without accounting for multiplicity can make small p-values misleading. Good reporting describes the study design and analysis, gives the p-value and relevant effect estimates with uncertainty intervals, and explains the result in context. Depending on the question, researchers may also use estimation, Bayesian methods, likelihood-based measures, or decision-theoretic approaches.