Hypothesis testing and p-value

When we want to check whether an observed effect is “real” or the result of chance, we formulate a null hypothesis $H_0$ (“there is no effect”) and an alternative hypothesis $H_1$. We compute the probability of obtaining a result at least as extreme as the one observed if $H_0$ were true: this number is the $p$-value.

• small $p$-value (conventionally $<0,05$) $\Rightarrow$ we reject $H_0$ (the effect is “statistically significant”).
• large $p$-value $\Rightarrow$ the data are compatible with $H_0$: we cannot conclude that there is no effect, only that the sample is not enough to reveal it.

Warning — Typical traps

$p<0,05$ does not mean ”$95\%$ probability that the effect is true”: it is the conditional probability $P(\text{data}\mid H_0)$, not $P(H_0\mid \text{data})$. Confusing them is the Bayesian muddle denounced in every course on statistical methodology. The distinction is the same one we saw in diagnostic tests (precision vs sensitivity).

Links

Topics: Probabilita
Concepts: Ipotesi nulla · P value · Probabilita condizionata · Test di ipotesi
Skills: Interpretare grafico · Stimare