Hypothesis testing and p-values

Is this coin fair? Does the new drug actually work, or did we get lucky? Hypothesis testing is the formal way to decide whether data is real evidence or just noise.

Assume nothing happened

Every test starts from a skeptic's default, the null hypothesis — "no effect, no difference, nothing going on." You then ask: if the null were true, how surprising is the data I actually saw?

The bell is the range of results you'd expect from pure chance if the null held. Your observed result lands somewhere on it.

The p-value

The p-value is the shaded tail: the probability of getting a result at least as extreme as yours, assuming the null is true.

• A small p-value (by convention, below 0.05) means your data would be rare under the null — so you doubt the null and call the result statistically significant.
• A large p-value means the data is perfectly consistent with chance — no evidence of an effect.

What a p-value is not

This is where careers and headlines go wrong:

• It is not the probability the null is true.
• It says nothing about how big or important the effect is — only how surprising the data is. With enough data, a trivial effect can be "significant."
• "Not significant" means absence of evidence, not evidence of absence.

Why this matters

Hypothesis tests gatekeep medicine, science, and A/B testing — deciding which effects are real enough to act on. But the same tool, misused (p-hacking, ignoring effect size), produces false findings. Understanding what a p-value genuinely claims is the difference between using statistics and being fooled by it.