Variability as a better characterization of Shannon entropy

TL;DR

This paper reinterprets Shannon entropy as a measure of variability within a distribution, clarifying its physical meaning and linking it to classical and quantum entropies in statistical mechanics.

Contribution

It introduces a novel perspective by characterizing Shannon entropy as variability, providing clearer understanding and connections to physical entropy concepts.

Findings

Shannon entropy measures the variability of elements in a distribution.

It is the only continuous and linear indicator of uncertainty.

Applying this view links Shannon entropy to Boltzmann, Gibbs, and von Neumann entropies.

Abstract

The Shannon entropy, one of the cornerstones of information theory, is widely used in physics, particularly in statistical mechanics. Yet its characterization and connection to physics remain vague, leaving ample room for misconceptions and misunderstanding. We will show that the Shannon entropy can be fully understood as measuring the variability of the elements within a given distribution: it characterizes how much variation can be found within a collection of objects. We will see that it is the only indicator that is continuous and linear, that it quantifies the number of yes/no questions (i.e. bits) that are needed to identify an element within the distribution, and we will see how applying this concept to statistical mechanics in different ways leads to the Boltzmann, Gibbs and von Neumann entropies.