Dimension-free Information Concentration via Exp-Concavity

TL;DR

This paper demonstrates that for exp-concave log-concave distributions, information concentration can be made independent of dimension, improving understanding in learning theory and enabling stronger probabilistic bounds.

Contribution

It proves dimension-free information concentration for exp-concave distributions using a novel application of the variance Brascamp-Lieb inequality.

Findings

Information content concentrates around differential entropy with dimension independence.

The result applies to high-probability bounds in learning theory.

Provides a new perspective on the role of exp-concavity in information concentration.

Abstract

Information concentration of probability measures have important implications in learning theory. Recently, it is discovered that the information content of a log-concave distribution concentrates around their differential entropy, albeit with an unpleasant dependence on the ambient dimension. In this work, we prove that if the potentials of the log-concave distribution are exp-concave, which is a central notion for fast rates in online and statistical learning, then the concentration of information can be further improved to depend only on the exp-concavity parameter, and hence, it can be dimension independent. Central to our proof is a novel yet simple application of the variance Brascamp-Lieb inequality. In the context of learning theory, our concentration-of-information result immediately implies high-probability results to many of the previous bounds that only hold in expectation.