Entropy Concentration and the Empirical Coding Game

TL;DR

This paper presents strong entropy concentration theorems that unify and extend classical results, clarifying the relationship between prior and constrained distributions in maximum entropy inference, and linking it to a game-theoretic perspective.

Contribution

It introduces two strong entropy concentration theorems that generalize existing theorems and connect entropy concentration with a game-theoretic interpretation of maximum entropy inference.

Findings

Theorems unify and extend classical entropy concentration results.

Exact characterization of the closeness between prior and constrained distributions.

Establishes a link between entropy concentration and game-theoretic maximum entropy inference.

Abstract

We give a characterization of Maximum Entropy/Minimum Relative Entropy inference by providing two `strong entropy concentration' theorems. These theorems unify and generalize Jaynes' `concentration phenomenon' and Van Campenhout and Cover's `conditional limit theorem'. The theorems characterize exactly in what sense a prior distribution Q conditioned on a given constraint, and the distribution P, minimizing the relative entropy D(P ||Q) over all distributions satisfying the constraint, are `close' to each other. We then apply our theorems to establish the relationship between entropy concentration and a game-theoretic characterization of Maximum Entropy Inference due to Topsoe and others.