A Lower Bound on the Relative Entropy with Respect to a Symmetric Probability

TL;DR

This paper establishes a lower bound on the relative entropy between two probability measures on the real line when one is symmetric, relating it to the mean and variance of the other measure, with equality characterizing the measures.

Contribution

It provides a new lower bound on the relative entropy with respect to a symmetric measure, linking it to the measure's mean and variance, and characterizes the equality case.

Findings

The lower bound is valid for measures with finite first moment.

Equality holds if and only if the measures are identical.

The bound relates relative entropy to the mean and second moment of the measure.

Abstract

Let $\rho$ and $\mu$ be two probability measures on $\mathbb{R}$ which are not the Dirac mass at $0$. We denote by $H(\mu|\rho)$ the relative entropy of $\mu$ with respect to $\rho$. We prove that, if $\rho$ is symmetric and $\mu$ has a finite first moment, then \[ H(\mu|\rho)\geq \frac{\displaystyle{(\int_{\mathbb{R}}z\,d\mu(z))^2}}{\displaystyle{2\int_{\mathbb{R}}z^2\,d\mu(z)}}\,,\] with equality if and only if $\mu=\rho$.