A Generalization of Relative Entropy to Count Vectors and its Concentration Property

TL;DR

This paper introduces a new generalized relative entropy for count vectors, demonstrating a concentration phenomenon under linear constraints and extending results to probabilistic settings and general convex sets.

Contribution

It generalizes relative entropy to non-negative vectors with sums greater than one and establishes a concentration property in a combinatorial setting without probabilistic assumptions.

Findings

Concentration phenomenon for the generalized relative entropy under linear constraints.

Extensions of concentration results to probabilistic formulations.

Simplifications and improvements over previous work, including dualization and relationships to KL-divergence.

Abstract

We introduce a new generalization of relative entropy to non-negative vectors with sums $\gt 1$. We show in a purely combinatorial setting, with no probabilistic considerations, that in the presence of linear constraints defining a convex polytope, a concentration phenomenon arises for this generalized relative entropy, and we quantify the concentration precisely. We also present a probabilistic formulation, and extend the concentration results to it. In addition, we provide a number of simplifications and improvements to our previous work, notably in dualizing the optimization problem, in the concentration with respect to $\ell_{\infty}$ distance, and in the relationship to generalized KL-divergence. A number of our results apply to general compact convex sets, not necessarily polyhedral.