Information in probability: Another information-theoretic proof of a finite de Finetti theorem

TL;DR

This paper presents a new information-theoretic proof of a finite de Finetti theorem, providing bounds on relative entropy for exchangeable sequences and connecting probability theory with statistical mechanics principles.

Contribution

It introduces a novel proof of the finite de Finetti theorem using information theory, with explicit entropy bounds and a connection to the Gibbs conditioning principle.

Findings

Derived an upper bound on relative entropy for exchangeable sequences.

Connected de Finetti's theorem to the Gibbs conditioning principle.

Utilized the method of types for technical estimates.

Abstract

We recall some of the history of the information-theoretic approach to deriving core results in probability theory and indicate parts of the recent resurgence of interest in this area with current progress along several interesting directions. Then we give a new information-theoretic proof of a finite version of de Finetti's classical representation theorem for finite-valued random variables. We derive an upper bound on the relative entropy between the distribution of the first $k$ in a sequence of $n$ exchangeable random variables, and an appropriate mixture over product distributions. The mixing measure is characterised as the law of the empirical measure of the original sequence, and de Finetti's result is recovered as a corollary. The proof is nicely motivated by the Gibbs conditioning principle in connection with statistical mechanics, and it follows along an appealing sequence of steps. The technical estimates required for these steps are obtained via the use of a collection of combinatorial tools known within information theory as `the method of types.'