Properties of entropy

Three properties capture the behaviour of entropy: it is always non-negative, it has a well-defined maximum, and it adds up for independent variables.

Property — Fundamental properties

• $H(X)\ge 0$, with $=0$ if and only if $X$ is constant;
• $H(X)\le \log_2 n$, with $=$ if and only if $X$ is uniform (the uniform maximises entropy, intuitively: maximum ignorance);
• additivity under independence: $H(X,Y) = H(X) + H(Y)$ if $X,Y$ are independent.

The second property is the rigorous version of the observation about the die: among all distributions over $n$ outcomes, the uniform one is the most unpredictable and its entropy $\log_2 n$ is the maximum possible.

Links

Topics: Distribuzioni probabilita
Concepts: Entropia
Skills: Dimostrare