From Physics to Statistics: A Simple Route to Exponential Families via Maximum Entropy

TL;DR

This paper presents a straightforward, self-contained derivation of exponential families from maximum entropy principles, making the concept more accessible in statistics education.

Contribution

It offers a simple derivation of exponential families from maximum entropy without advanced prerequisites or complex optimization techniques.

Findings

Exponential families with a general base maximize entropy given fixed expectations.

Exponential families with a uniform base maximize standard entropy under the same constraints.

Abstract

Exponential families form the backbone of modern statistics and machine learning, but textbooks seldom derive them from first principles in an accessible way. Although minimal sufficiency and the principle of maximum entropy, originating in physics, provide core motivation, they are often presented as technical and requiring advanced prerequisites. Here, a short, self-contained derivation of exponential families based on maximum entropy is presented that is straightforward to carry out, requires only a modest background in information entropy, and avoids technicalities like constrained optimisation. Two propositions are demonstrated in this fashion: i) exponential families with a general base maximise information entropy with respect to that base subject to fixed expectations of canonical statistics, and ii) exponential families with a uniform base maximise standard information entropy under the same constraints. Maximum entropy therefore provides a principled foundation for exponential families with minimal prerequisites, highlighting the value of teaching entropy in statistics courses.