Entropy Concentration and the Empirical Coding Game
Peter Grunwald

TL;DR
This paper presents strong entropy concentration theorems that unify and extend classical results, clarifying the relationship between prior and constrained distributions in maximum entropy inference, and linking it to a game-theoretic perspective.
Contribution
It introduces two strong entropy concentration theorems that generalize existing theorems and connect entropy concentration with a game-theoretic interpretation of maximum entropy inference.
Findings
Theorems unify and extend classical entropy concentration results.
Exact characterization of the closeness between prior and constrained distributions.
Establishes a link between entropy concentration and game-theoretic maximum entropy inference.
Abstract
We give a characterization of Maximum Entropy/Minimum Relative Entropy inference by providing two `strong entropy concentration' theorems. These theorems unify and generalize Jaynes' `concentration phenomenon' and Van Campenhout and Cover's `conditional limit theorem'. The theorems characterize exactly in what sense a prior distribution Q conditioned on a given constraint, and the distribution P, minimizing the relative entropy D(P ||Q) over all distributions satisfying the constraint, are `close' to each other. We then apply our theorems to establish the relationship between entropy concentration and a game-theoretic characterization of Maximum Entropy Inference due to Topsoe and others.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsStatistical Mechanics and Entropy · Complex Systems and Time Series Analysis
