Minimum KL-divergence on complements of $L_1$ balls

TL;DR

This paper investigates the minimal Kullback-Leibler divergence needed for distributions to be a certain total variation distance away from a fixed distribution, providing a reverse Pinsker inequality with applications to large deviations.

Contribution

It establishes an upper bound on the infimum of KL-divergence over distributions at a fixed TV distance, introducing a reverse Pinsker inequality with explicit constants.

Findings

D*(P,ε) ≤ 1/2 * ε^2 + O(ε^3) for balanced distributions

Provides a reverse Pinsker inequality with explicit bounds

Applications to large deviations and structural insights

Abstract

Pinsker's widely used inequality upper-bounds the total variation distance $||P-Q||_1$ in terms of the Kullback-Leibler divergence $D(P||Q)$. Although in general a bound in the reverse direction is impossible, in many applications the quantity of interest is actually $D^*(P,\eps)$ --- defined, for an arbitrary fixed $P$, as the infimum of $D(P||Q)$ over all distributions $Q$ that are $\eps$-far away from $P$ in total variation. We show that $D^*(P,\eps)\le C\eps^2 + O(\eps^3)$, where $C=C(P)=1/2$ for "balanced" distributions, thereby providing a kind of reverse Pinsker inequality. An application to large deviations is given, and some of the structural results may be of independent interest. Keywords: Pinsker inequality, Sanov's theorem, large deviations