Generalized Entropy Concentration for Counts

TL;DR

This paper extends the entropy concentration phenomenon to non-negative integral vectors with linear constraints, introducing a generalized entropy measure that supports concentration analysis beyond traditional MaxEnt applications.

Contribution

It introduces a generalized entropy measure and demonstrates entropy concentration for integral vectors under linear sum constraints, expanding MaxEnt applicability.

Findings

Entropy concentration holds for integral vectors with sum constraints.

Non-asymptotic bounds on concentration are established.

Generalized entropy extends MaxEnt to broader problems.

Abstract

The phenomenon of entropy concentration provides strong support for the maximum entropy method, MaxEnt, for inferring a probability vector from information in the form of constraints. Here we extend this phenomenon, in a discrete setting, to non-negative integral vectors not necessarily summing to 1. We show that linear constraints that simply bound the allowable sums suffice for concentration to occur even in this setting. This requires a new, `generalized' entropy measure in which the sum of the vector plays a role. We measure the concentration in terms of deviation from the maximum generalized entropy value, or in terms of the distance from the maximum generalized entropy vector. We provide non-asymptotic bounds on the concentration in terms of various parameters, including a tolerance on the constraints which ensures that they are always satisfied by an integral vector. Generalized entropy maximization is not only compatible with ordinary MaxEnt, but can also be considered an extension of it, as it allows us to address problems that cannot be formulated as MaxEnt problems.