Entropy of a die

With more than two outcomes the entropy grows, but it depends on how “unbalanced” the distribution is: a fair die is more unpredictable than a biased one.

Example — Six-sided die

$X$ uniform over $\{1,\ldots,6\}$: $H = -6\cdot\frac{1}{6}\log_2\frac{1}{6} = \log_2 6 \approx 2{,}585 \text{ bit}.$ On average about $2{,}6$ well-chosen yes / no questions are needed to guess the outcome.

Biased die with $P(6)=1/2$ and $P(1)=\cdots=P(5)=1/10$: $H = -\tfrac12\log_2\tfrac12 - 5\cdot\tfrac{1}{10}\log_2\tfrac{1}{10} = \tfrac12 + \tfrac12\log_2 10 \approx 2{,}16 \text{ bit}.$ Knowing that the $6$ is likely reduces the uncertainty: fewer bits to describe the average result.

The comparison illustrates a general principle: for a given number of outcomes, the uniform distribution is the one of maximum entropy; any imbalance in the probabilities lowers the entropy.

Links

Topics: Distribuzioni probabilita
Concepts: Entropia
Skills: Calcolare · Interpretare grafico