Finite-sample concentration of the empirical relative entropy around its mean

TL;DR

This paper proves that the empirical relative entropy from samples concentrates exponentially around its mean, with bounds tighter than previous work, confirming a recent conjecture and improving understanding of sample-based entropy estimates.

Contribution

It provides a sharper exponential concentration bound for empirical relative entropy, improving upon prior bounds and confirming a recent conjecture.

Findings

Exponential concentration bound with gamma distribution parameters

Improved bounds over previous polylogarithmic factors

Reduction from multinomial to binomial case for proof

Abstract

In this note, we show that the relative entropy of an empirical distribution of $n$ samples drawn from a set of size $k$ with respect to the true underlying distribution is exponentially concentrated around its expectation, with central moment generating function bounded by that of a gamma distribution with shape $2k$ and rate $n/2$. This improves on recent work of Bhatt and Pensia (arXiv 2021) on the same problem, who showed such a similar bound with an additional polylogarithmic factor of $k$ in the shape, and also confirms a recent conjecture of Mardia et al. (Information and Inference 2020). The proof proceeds by reducing the case $k>3$ of the multinomial distribution to the simpler case $k=2$ of the binomial, for which the desired bound follows from standard results on the concentration of the binomial.