Explicit Bounds for Entropy Concentration under Linear Constraints

TL;DR

This paper provides explicit, non-asymptotic bounds on the sample size needed for entropy concentration under linear constraints, accounting for approximation tolerances and using novel probabilistic techniques.

Contribution

It introduces the first non-asymptotic, explicit bounds for entropy concentration with constraints holding approximately, using the Berry-Esseen theorem for the first time in this context.

Findings

Derived explicit bounds on sample size for entropy concentration

Accounted for approximate satisfaction of constraints

Applied Berry-Esseen theorem to multidimensional setting

Abstract

Consider the set of all sequences of $n$ outcomes, each taking one of $m$ values, that satisfy a number of linear constraints. If $m$ is fixed while $n$ increases, most sequences that satisfy the constraints result in frequency vectors whose entropy approaches that of the maximum entropy vector satisfying the constraints. This well-known "entropy concentration" phenomenon underlies the maximum entropy method. Existing proofs of the concentration phenomenon are based on limits or asymptotics and unrealistically assume that constraints hold precisely, supporting maximum entropy inference more in principle than in practice. We present, for the first time, non-asymptotic, explicit lower bounds on $n$ for a number of variants of the concentration result to hold to any prescribed accuracies, with the constraints holding up to any specified tolerance, taking into account the fact that allocations of discrete units can satisfy constraints only approximately. Again unlike earlier results, we measure concentration not by deviation from the maximum entropy value, but by the $\ell_1$ and $\ell_2$ distances from the maximum entropy-achieving frequency vector. One of our results holds independently of the alphabet size $m$ and is based on a novel proof technique using the multi-dimensional Berry-Esseen theorem. We illustrate and compare our results using various detailed examples.