Two-Moment Inequalities for R\'enyi Entropy and Mutual Information

TL;DR

This paper introduces new two-moment inequalities for Re9nyi entropy and mutual information, providing tighter bounds and improved methods for information measure estimation using moment-based techniques.

Contribution

It presents novel two-moment inequalities for Re9nyi entropy and a new approach to upper bounding mutual information via variance-based integrals.

Findings

Bounds on Re9nyi entropy improve with two moments.

New upper bounds on mutual information relate to variance of conditional density.

Method enhances estimation accuracy with modest complexity increase.

Abstract

This paper explores some applications of a two-moment inequality for the integral of the $r$-th power of a function, where $0 < r< 1$. The first contribution is an upper bound on the R\'{e}nyi entropy of a random vector in terms of the two different moments. When one of the moments is the zeroth moment, these bounds recover previous results based on maximum entropy distributions under a single moment constraint. More generally, evaluation of the bound with two carefully chosen nonzero moments can lead to significant improvements with a modest increase in complexity. The second contribution is a method for upper bounding mutual information in terms of certain integrals with respect to the variance of the conditional density. The bounds have a number of useful properties arising from the connection with variance decompositions.

Equations

$$0<r<1\text{and}p<\frac{1-r}{r}<q,$$

$$\mathopen{}\mathclose{{}\left(\int\!f^{r}(x)\,\mathrm{d}x\!}\right)^{\!\frac{1}{r}}\!\!\!\leq C\,\mathopen{}\mathclose{{}\left(\int\!|x|^{p}f(x)\,\mathrm{d}x\!}\right)^{\!\lambda}\!\mathopen{}\mathclose{{}\left(\int\!|x|^{q}f(x)\,\mathrm{d}x\!}\right)^{\!1-\lambda}\!\!,$$

$$\int \Vert y{\Vert }^s\mathrm{Var}(f_{Y|X}(y|X))\mathrm{d}y.$$

$$\|f\|_{p}=\mathopen{}\mathclose{{}\left(\int_{S}\mathopen{}\mathclose{{}\left|f(x)}\right|^{p}\,\mathrm{d}x}\right)^{\frac{1}{p}}.$$

$${\mu }_s(f)={\int }_S\Vert x{\Vert }^{s}f(x)\mathrm{d}x,$$

$$\Vert f{\Vert }_r$$

$$f(x)\ge 0$$

$${\mu }_{s_i}(f)\le m_i$$

$$f^{*}(x)=\bigg{(}\sum_{i=1}^{k}\nu^{*}_{i}\,\|x\|^{s_{i}}\bigg{)}^{\frac{1}{r-1}},\quad\text{for all $x\in S$}.$$

$$\Vert f^{\ast }{\Vert }_r^r=\Vert (f^{\ast }{)}^r{\Vert }_1=\Vert f^{\ast }(f^{\ast }{)}^{r-1}{\Vert }_1=\sum _{i=1}^k{\nu }_i^{\ast }{m}_i.$$

$$c_{r}(\nu,s)=\mathopen{}\mathclose{{}\left(\int_{0}^{\infty}\bigg{(}\sum_{i=1}^{k}\nu_{i}\,x^{s_{i}}\bigg{)}^{-\frac{r}{1-r}}\mathrm{d}x}\right)^{\frac{1-r}{r}},$$

$$\Vert f{\Vert }_r\le c_r(\nu ,s)\sum _{i=1}^k{\nu }_i{\mu }_{s_i}(f).$$

$$\Vert f{\Vert }_r^r$$

$$\le \Vert g^{-r}{\Vert }_{\frac{1}{1-r}}\Vert (gf{)}^r{\Vert }_{\frac{1}{r}}$$

$$=\Vert g^{\frac{-r}{1-r}}{\Vert }_1^{1-r}\Vert gf{\Vert }_1^r$$

$$\displaystyle=\bigg{(}c_{r}(\nu,s)\,\sum_{i=1}^{k}\nu_{i}\,\mu_{s_{i}}(f)\bigg{)}^{r},$$

$$\|f\|_{r}\leq\mathopen{}\mathclose{{}\left[\psi_{r}(p,q)}\right]^{\frac{1-r}{r}}\,[\mu_{p}(f)]^{\lambda}[\mu_{q}(f)]^{1-\lambda},$$

$$\displaystyle\psi_{r}(p,q)=\frac{1}{(q-p)}\widetilde{\mathrm{B}}\mathopen{}\mathclose{{}\left(\frac{r\lambda}{1-r},\frac{r(1-\lambda)}{1-r}}\right),$$

$$[c_{r}(\nu,s)]^{\frac{r}{1-r}}=\int_{0}^{\infty}\mathopen{}\mathclose{{}\left(\gamma^{1-\lambda}\,x^{p}+\gamma^{-\lambda}\,x^{q}}\right)^{-\frac{r}{1-r}}\,\mathrm{d}x.$$

$$[c_{r}(\nu,s)]^{\frac{r}{1-r}}=\frac{1}{(q-p)}\int_{0}^{\infty}\frac{u^{b-1}}{(1+u)^{a+b}}\,\mathrm{d}u=\frac{\mathrm{B}\mathopen{}\mathclose{{}\left(a,b}\right)}{(q-p)},$$

$$\|f\|_{r}\leq\mathopen{}\mathclose{{}\left(\frac{\mathrm{B}\mathopen{}\mathclose{{}\left(a,b}\right)}{q-p}}\right)^{\frac{1-r}{r}}\mathopen{}\mathclose{{}\left(\gamma^{1-\lambda}\mu_{p}(f)+\gamma^{-\lambda}\,\mu_{q}(f)}\right),$$

$$\gamma =\frac{\lambda {\mu }_q(f)}{(1-\lambda ){\mu }_p(f)},$$

$${\psi }_{1/2}(p,q)=\frac{\pi {\lambda }^{-\lambda }(1-\lambda {)}^{-(1-\lambda )}}{(q-p)\sin (\pi \lambda )},$$

$$\omega (S)=\mathrm{Vol}(B^n\cap \mathrm{cone}(S)),$$

$$\mathrm{cone}(S)=\{x\in {\mathbb{R}}^n:\text{tx∈S for some t>0}\}.$$

$$\omega (S)\le \omega ({\mathbb{R}}^n)=\frac{{\pi }^{\frac{n}{2}}}{\Gamma (\frac{n}{2}+1)},$$

$$\|f\|_{r}\leq\mathopen{}\mathclose{{}\left[\omega(S)\,\psi_{r}(p,q)}\right]^{\frac{1-r}{r}}\,[\mu_{np}(f)]^{\lambda}[\mu_{nq}(f)]^{1-\lambda},$$

$$g(y)=\frac{1}{n}{\int }_{{\mathbb{S}}^{n-1}}f(y^{\frac{1}{n}}u)\mathrm{d}\sigma (u),$$

$$\Vert f{\Vert }_r$$

$${\mu }_{ns}(f)$$

$$\|f\|^{r}_{r}=\int_{0}^{\infty}\int_{\mathbb{S}^{n-1}}\mathopen{}\mathclose{{}\left|f(tu)}\right|^{r}t^{n-1}\,\mathrm{d}\sigma(u)\,\mathrm{d}t.$$

$$\displaystyle\int_{\mathbb{S}^{n-1}}\mathopen{}\mathclose{{}\left|f(tu)}\right|^{r}\,\mathrm{d}\sigma(u)$$

$$\displaystyle=\int_{\mathbb{S}^{n-1}}\bm{1}_{\operatorname{cone}(S)}(u)\,\mathopen{}\mathclose{{}\left|f(tu)}\right|^{r}\,\mathrm{d}\sigma(u)$$

$$\displaystyle\overset{(a)}{\leq}\mathopen{}\mathclose{{}\left(\int_{\mathbb{S}^{n-1}}\!\bm{1}_{\operatorname{cone}(S)}(u)\,\mathrm{d}\sigma(u)\!}\right)^{\!1-r}\mathopen{}\mathclose{{}\left(\int_{\mathbb{S}^{n-1}}\mathopen{}\mathclose{{}\left|f(tu)}\right|\,\mathrm{d}\sigma(u)\!}\right)^{r}$$

Full text

I Introduction

Measures of entropy and information play a central role in applications throughout information theory, statistics, computer science, and statistical physics. In many cases, there is interest in understanding maximal properties of these measures over a given family of distributions. One example is given by the principle of maximum entropy, which originated in statistical mechanics and was introduced in broader context by Jaynes [1].

Entropy-moment inequalities can be used to describe properties of distributions characterized by moment constraints. Perhaps the most well known entropy-moment inequality follows from the fact that the Gaussian distribution maximizes differential entropy over all distributions with the same variance [2, Theorem 8.6.5]. This inequality leads to remarkably simple proofs for fundamental results in information theory and estimation theory.

A variety of entropy-moment inequalities have also been studied in the context of Rényi entropy [3, 4, 5, 6, 7], which is a generalization of Shannon entropy. Recent work has focused on the extremal distributions for the closely related Rényi divergence [8, 9, 10, 11, 12].

Another line of work focuses on relationships between measures of dissimilarity between probability distributions provided by the family of $f$-divergences [13, 14], which includes as special cases, the total variation distance, relative entropy (or Kullback-Leibler divergence), Rényi divergence, and chi-square divergence. One application of these results is to provide bounds for mutual information in terms of divergence measures that dominate relative entropy, such as the chi-square divergence; see e.g. [13, 15].

I-A Overview of results

The starting point of our analysis (Proposition 2) is an inequality for the integral of the $r$-th power of a function. Specifically, for any numbers $p,q,r$ with

[TABLE]

the following inequality holds:

[TABLE]

for all non-negative functions $f:\mathbb{R}_{+}\to\mathbb{R}_{+}$ where $C$ and $0<\lambda<1$ are given explicitly in terms of the tuple $(p,q,r)$. An extension to functions defined on an arbitrary subset of $\mathbb{R}^{n}$ is also provided (Proposition 3).

The remainder of the paper shows how this inequality can be used to provide bounds on information measures such as Rényi entropy and mutual information. Some useful properties of the bounds include:

• •

Simplicity: Beyond the existence of a density, these bounds do not require further regularity conditions such as boundedness or sub-exponential tails. As a consequence, these bounds can be applied under relatively mild technical assumptions.

• •

Tightness: For some applications, the bounds can provide an accurate characterization of the underlying information measures. For example, a special case of Proposition 9 in this paper played a key role in the author’s recent work [16, 17, 18], where it was used to bound the relative entropy between low-dimensional projections of a random vector and a Gaussian approximation.

• •

Geometric Interpretation: Our bounds on the mutual information between random variables $X$ and $Y$ can be expressed in terms of the variance of the conditional density of $Y$ given $X$. Specifically, the bounds depend on integrals of the form:

[TABLE]

For $s=0$, this integral is the expected squared $L^{2}$ distance between the conditional density $f_{Y|X}$ and the marginal density $f_{Y}$.

The paper is organized as follows: Section II provides integral inequalities for nonnegative functions; Section III gives bounds on Rényi entropy of orders less than one; and Section IV provides bounds on mutual information.

II Moment Inequalities

Throughout this section, we assume that $f$ is a real-valued Lebesgue measurable function defined on a measurable subset $S$ of $\mathbb{R}^{n}$. For any positive number $p$, the function $\|\cdot\|_{p}$ is defined according to

[TABLE]

Recall that for $0<p<1$, the function $\|\cdot\|_{p}$ is not a norm because it does not satisfy the triangle inequality. The $s$-th moment of $f$ is defined according to

[TABLE]

where $\|\cdot\|$ denotes the standard Euclidean norm on vectors.

II-A Multiple Moments

Consider the following optimization problem:

[TABLE]

For $r\in(0,1)$ this is a convex optimization problem because $\|\cdot\|_{r}^{r}$ is concave and the moment constraints are linear. By standard theory in convex optimization (see e.g., [19]), it can be shown that if the problem is feasible and the maximum is finite, then the maximizer has the form

[TABLE]

The parameters $\nu^{*}_{1},\cdots,\nu^{*}_{k}$ are nonnegative and the $i$-th moment constraint holds with equality for all $i$ such that $\nu^{*}_{i}$ is strictly positive, that is $\nu^{*}_{i}>0\implies\mu_{s_{i}}(f^{*})=m_{i}$. Consequently, the maximum can be expressed in terms of a linear combination of the moments:

[TABLE]

For the purposes of this paper, is it is useful to consider a relative inequality in terms of the moments of the function itself. Given a number $0<r<1$ and vectors $s\in\mathbb{R}^{k}$ and $\nu\in\mathbb{R}_{+}^{k}$ the function $c_{r}(\nu,s)$ is defined according to

[TABLE]

if the integral exists. Otherwise, $c_{r}(\nu,s)$ is defined to be positive infinity. It can be verified that $c_{r}(\nu,s)$ is finite provided that there exists $i,j$ such that $\nu_{i}$ and $\nu_{j}$ are strictly positive and $s_{i}<(1-r)/r<s_{j}$.

The following result can be viewed as a consequence of the constrained optimization problem described above. We provide a different and very simple proof that depends only on Hölder’s inequality.

Proposition 1**.**

Let $f$ be a nonnegative Lebesgue measurable function defined on the positive reals $\mathbb{R}_{+}$. For any number $0<r<1$ and vectors $s\in\mathbb{R}^{k}$ and $\nu\in\mathbb{R}_{+}^{k}$, we have

[TABLE]

Proof.

Let $g(x)=\sum_{i=1}^{k}\nu_{i}\,x^{s_{i}}$. Then, we have

[TABLE]

where second step follows from Hölder’s inequality with conjugate exponents $1/(1-r)$ and $1/r$. ∎

II-B Two Moments

The next result follows from Proposition 1 for the case of two moments.

Proposition 2**.**

Let $f$ be a nonnegative Lebesgue measureable function defined on the positive reals $\mathbb{R}_{+}$. For any numbers $p,q,r$ with $0<r<1$ and $p<1/r-1<q$, we have

[TABLE]

where $\lambda=(q+1-1/r)/(q-p)$ and

[TABLE]

where $\widetilde{B}(a,b)=\mathrm{B}(a,b)(a+b)^{a+b}a^{-a}b^{-b}$ and $\mathrm{B}(a,b)$ is the Beta function.

Proof.

Letting $s=(p,q)$ and $\nu=(\gamma^{1-\lambda},\gamma^{-\lambda})$ with $\lambda>0$, we have

[TABLE]

Making the change of variable $x\mapsto(\gamma u)^{\frac{1}{q-p}}$ leads to

[TABLE]

where $a=\frac{r}{1-r}\lambda$ and $b=\frac{r}{1-r}(1-\lambda)$ and the second step follows from the integral representation of the Beta function [20, Eq. (1.1.19)]. Therefore, by Proposition 1, the inequality

[TABLE]

holds for all $\gamma>0$. Evaluating this inequality with

[TABLE]

leads to the stated result. ∎

The special case $r=1/2$ admits the simplified expression

[TABLE]

where we have used Euler’s reflection formula for the Beta function [20, Theorem 1.2.1].

Next, we consider an extension of Proposition 2 for functions defined on $\mathbb{R}^{n}$. Given any measurable subset $S$ of $\mathbb{R}^{n}$ we define

[TABLE]

where $B^{n}=\{u\in\mathbb{R}^{n}:\|u\|\leq 1\}$ is the $n$-dimensional Euclidean ball of radius one and

[TABLE]

The function $\omega(S)$ is proportional to the surface measure of the projection of $S$ on the Euclidean sphere and satisfies

[TABLE]

for all $S\subseteq\mathbb{R}^{n}$. Note that $\omega(\mathbb{R}_{+})=1$ and $\omega(\mathbb{R})=2$.

Proposition 3**.**

Let $f$ be a nonnegative Lebesgue measurable function defined on a subset $S$ of $\mathbb{R}^{n}$. For any numbers $p,q,r$ with $0<r<1$ and $p<1/r-1<q$, we have

[TABLE]

where $\lambda=(q+1-1/r)/(q-p)$ and $\psi_{r}(p,q)$ is given by (1).

Proof.

Let $f$ be extended to $\mathbb{R}^{n}$ using the rule $f(x)=0$ for all $x$ outside of $S$ and let $g:\mathbb{R}_{+}\to\mathbb{R}_{+}$ be defined according to

[TABLE]

where $\mathbb{S}^{n-1}=\{u\in\mathbb{R}^{n}\,:\,\|u\|=1\}$ is the Euclidean sphere of radius one and $\sigma(u)$ is the surface measures of the sphere. We will show that

[TABLE]

Then, the stated inequality then follows from applying Proposition 2 to the function $g$.

In order to prove (5), we begin with a transformation into polar coordinates:

[TABLE]

Letting $\bm{1}_{\operatorname{cone}(S)}(x)$ denote the indicator function of the set $\operatorname{cone}(S)$, the integral over the sphere can be bounded using:

[TABLE]

where: (a) follows from Hölder’s inequality with conjugate exponents $\frac{1}{1-r}$ and $\frac{1}{r}$; and (b) follows from the definition of $g$ and the fact that

[TABLE]

Plugging (8) back into (7) and then making the change of variables $t\to y^{\frac{1}{n}}$ yields

[TABLE]

The proof of (6) follows along similar lines. We have

[TABLE]

where (a) follows from a transformation into polar coordinates and (b) follows form the change of variable $t\mapsto y^{\frac{1}{n}}$. ∎

III Rényi Entropy Bounds

Let $X$ be a random vector that has a density $f(x)$ with respect to Lebesgue measure on $\mathbb{R}^{n}$. The differential Rényi entropy of order $r\in(0,1)\cup(1,\infty)$ is defined according to [2]:

[TABLE]

The Rényi entropy is continuous and non-increasing in $r$. If the support set $S=\{x\in\mathbb{R}^{n}:f(x)>0\}$ has finite measure then the limit as $r$ converges to zero is given by $h_{0}(X)=\log\operatorname{Vol}(S)$. If the support does not have finite measure then $h_{r}(X)$ increases to infinity as $r$ decreases to zero. The case $r=1$ is given by the Shannon differential entropy:

[TABLE]

Given a random variable $X$ that is not identically zero and numbers $p,q,r$ with $0<r<1$ and $p<1/r-1<q$, we define the function

[TABLE]

where $\lambda=(q+1-1/r)/(q-p)$.

The next result, which follows directly from Proposition 3, provides an upper bound on the Rényi entropy.

Proposition 4**.**

Let $X$ be a random vector with a density on $\mathbb{R}^{n}$. For any numbers $p,q,r$ with $0<r<1$ and $p<1/r-1<q$, the Rényi entropy satisfies

[TABLE]

where $\omega(S)$ is defined in (3) and $\psi_{r}(p,q)$ is defined in (1).

Proof.

This result follows immediately from Proposition 3 and the definition of Rényi entropy. ∎

The relationship between Proposition 4 and previous results depends on whether the moment $p$ is equal to zero:

• •

One-moment inequalities: If $p=0$ then there exists a distribution such that (9) holds with equality. This is because the zero-moment constraint ensures that the function that maximizes the Rényi entropy integrates to one. In this case, Proposition 4 is equivalent to previous results that focused on distributions that maximize Rényi entropy subject to a single moment constraint [3, 4, 5]. With some abuse of terminology we refer to these bounds as one-moment inequalities111A more accurate name would be two-moment inequalities under the constraint that one of the moments is the zeroth moment..

• •

Two-moment inequalities: If $p\neq 0$ then the right-hand side of (9) corresponds to the Rényi entropy of a non-negative function that might not integrate to one. Nevertheless, the expression provides an upper bound on the Rényi entropy for any density with the same moments. We refer to the bounds obtained using a general pair $(p,q)$ as two-moment inequalities.

The contribution of two-moment inequalities is that they lead to tighter bounds. To quantify the tightness, we define $\Delta_{r}(X;p,q)$ to be the gap between the right-hand side and left-hand side of (9) corresponding to the pair $(p,q)$, that is

[TABLE]

The gaps corresponding to the optimal two-moment and one-moment inequalities are defined according to:

[TABLE]

III-A Some consequences of these bounds

By Lyapunov’s inequality, the mapping s\mapsto\frac{1}{s}\log\mathbb{E}\mathopen{}\mathclose{{}\left[|X|^{s}}\right] is nondecreasing on $[0,\infty)$ and thus

[TABLE]

In other words, the case $p=0$ provides an upper bound on $L_{r}(X;p,q)$ for nonnegative $p$. Alternatively, we also have the lower bound

[TABLE]

which follows from the convexity of \log\mathbb{E}\mathopen{}\mathclose{{}\left[|X|^{s}}\right].

A useful property of $L_{r}(X;p,q)$ is that it is additive with respect to the product of independent random variables. Specifically, if $X$ and $Y$ are independent, then

[TABLE]

One consequence is that multiplication by a bounded random variable cannot increase the Rényi entropy by an amount that exceeds the gap of the two-moment inequality with nonnegative moments.

Proposition 5**.**

Let $Y$ be a random vector on $\mathbb{R}^{n}$ with finite Rényi entropy of order $0<r<1$, and let $X$ be an independent random variable that satisfies $0<X\leq t$. Then,

[TABLE]

for all $0<p<1/r-1<q$.

Proof.

Let $Z=XY$ and let $S_{Z}$ and $S_{Y}$ denote the support sets of $Z$ and $Y$, respectively. The assumption that $X$ is non-negative means that $\operatorname{cone}(S_{Z})=\operatorname{cone}(S_{Y})$. We have

[TABLE]

where: (a) follows from Proposition 4; (b) follows from (12) and the definition of $\Delta_{r}(Y;p,q)$, and (c) follows from (10) and the assumption $|X|\leq t$. Finally, recalling that $h_{r}(tY)=h_{r}(Y)+n\log t$ completes the proof. ∎

III-B Example with lognormal distribution

If $W\sim\mathcal{N}(\mu,\sigma^{2})$ then the random variable $X=\exp(W)$ has a lognormal distribution with parameters $(\mu,\sigma^{2})$. The Rényi entropy is given by

[TABLE]

and the logarithm of the $s$-th moment is given by

[TABLE]

With a bit of work, it can be shown that the gap of the optimal two-moment inequality does not depend on the parameters $(\mu,\sigma^{2})$ and is given by

[TABLE]

The details of this derivation are given in Appendix B-A. Meanwhile, the gap of the optimal one-moment inequality is given by

[TABLE]

The functions $\Delta_{r}(X)$ and $\widetilde{\Delta}_{r}(X)$ are illustrated in Figure 1 as a function of $r$ for various $\sigma^{2}$. The function $\Delta_{r}(X)$ is bounded uniformly with respect to $r$ and converges to zero as $r$ increases to one. The tightness of the two-moment inequality in this regime follows from the fact that the lognormal distribution maximizes Shannon entropy subject to a constraint on \mathbb{E}\mathopen{}\mathclose{{}\left[\log X}\right]. By contrast, the function $\widetilde{\Delta}_{r}(X)$ varies with the parameter $\sigma^{2}$. For any fixed $r\in(0,1)$, it can be shown that $\widetilde{\Delta}_{r}(X)$ increases to infinity if $\sigma^{2}$ converges to zero or infinity.

III-C Example with multivariate Gaussian distribution

Next, we consider the case where $Y\sim\mathcal{N}(0,I_{n})$ is an $n$-dimensional Gaussian vector with mean zero and identity covariance. The Rényi entropy is given by

[TABLE]

and the $s$-th moment of the magnitude $\|X\|$ is given by

[TABLE]

As the dimension $n$ increases, it can be shown that the gap of the optimal two-moment inequality converges to the gap for the lognormal distribution. The proof of the following result is given in Appendix B-C.

Proposition 6**.**

If $Y\sim\mathcal{N}(0,I_{n})$ then,

[TABLE]

where $X$ has a lognormal distribution.

The functions $\Delta_{r}(Y)$ and $\widetilde{\Delta}_{r}(Y)$ are illustrated in Figure 2. Both functions are increasing in the dimension $n$. However, while $\Delta_{r}(Y)$ converges to a finite limit, $\widetilde{\Delta}_{r}(Y)$ increases without bound. For any fixed integer $n$, it can be shown that both $\Delta_{r}(Y)$ and $\widetilde{\Delta}_{r}(Y)$ converge to zero as $r$ increases to one. This behavior follows from the fact that the Gaussian distribution is the maximum entropy distribution for Shannon entropy under a second moment constraint.

III-D Inequalities for differential entropy

Proposition 4 can also be used to recover some known inequalities for differential entropy by considering the limiting behavior as $r$ converges to one. For example, it is well known that the differential entropy of an $n$-dimensional random vector $X$ with finite second moment satisfies

[TABLE]

with equality if and only if the entries of $X$ are i.i.d. zero-mean Gaussian. A generalization of this result in terms an arbitrary positive moment is given by

[TABLE]

for all $s>0$. Note that (15) corresponds to the case $s=2$.

Inequality (16) can be proved as an immediate consequence of Proposition 4 and the fact that $h_{r}(X)$ is non-increasing in $r$. Using properties of the beta function given in Appendix A, it is straightforward to verify that

[TABLE]

Combining this result with Proposition 4 and (10) leads to

[TABLE]

Using (4) and making the substitution $s=nq$ leads to (16).

Another example follows from the fact that the lognormal distribution maximizes the differential entropy of a positive random variable $X$ subject to constraints on the mean and variance of $\log(X)$, and hence

[TABLE]

with equality if and only if $X$ is lognormal. In Appendix B-D, it is shown how this inequality can be proved using our two-moment inequalities, by studying the behavior as both $p$ and $q$ converge to zero as $r$ increases to one.

IV Mutual Information Bounds

IV-A Relative entropy and chi-square divergence

Let $P$ and $Q$ be distributions defined on common probability space that that have densities $p$ and $q$ with respect to a dominated measure $\mu$. The relative entropy (or Kullback–Leibler) divergence is defined according to

[TABLE]

and the chi-square divergence is defined according to,

[TABLE]

The chi-square divergence is equal to the squared $L_{2}$ distance between the densities scaled densities $p/\sqrt{q}$ and $\sqrt{q}$. The chi-square can also be interpreted as the first non-zero term in the power series expansion of the relative entropy [13, Lemma 4]. More generally, the chi-square provides an upper bound on the relative entropy, via

[TABLE]

The proof of this inequality follows straightforwardly from Jensen’s inequality and the concavity of the logarithm; see e.g., [21, Theorem 5].

Given a random pair $(X,Y)$ the mutual information between $X$ and $Y$ is defined according to

[TABLE]

From (18), we see the the mutual information can always be upper bounded using

[TABLE]

The next section provides bounds on the mutual information that can improve upon this inequality.

IV-B Mutual information and variance of conditional density

Let $(X,Y)$ be a random pair such that the conditional distribution $Y$ given $X$ has a density $f_{Y|X}(y|x)$ with respect to Lebesgue measure on $\mathbb{R}^{n}$. Note that the marginal density of $Y$ is given by f_{Y}(y)=\mathbb{E}\mathopen{}\mathclose{{}\left[f_{Y|X}(y|X)}\right]. To simplify notation, we will write $f(y|x)$ and $f(y)$ where the subscripts are implicit. The support set of $Y$ denoted by $S_{Y}$.

The measure of the dependence between $X$ and $Y$ that is used in our bounds can be understood in terms of the variance of the conditional density. For each $y$, the conditional density $f(y|X)$ evaluated with a random realization of $X$ is a random variable. The variance of this random variable is given by

[TABLE]

where we have used the fact that the marginal density $f(y)$ is the expectation of $f(y|X)$. The $s$-th moment of the variance of the conditional density is defined according to

[TABLE]

The function $V_{s}(Y|X)$ is nonnegative and equal to zero if and only if $X$ and $Y$ are independent.

For $t\in(0,1]$ the function $\kappa(t)$ is defined according to

[TABLE]

Properties of this function are given in Appendix C, where it is shown that $1/(e\,t)<\kappa(t)\leq 1/t$ with equality on the right when $t=1$.

We are now ready to give the main results of this section, which are bounds on the mutual information. We begin with a general upper bound in terms of the variance of the conditional density.

Proposition 7**.**

For any $0<t\leq 1$, the mutual information satisfies

[TABLE]

Proof.

We use the following series of inequalities:

[TABLE]

where: (a) follows from the definition of mutual information; (b) follows from (18); and (c) follows from Bayes’ rule, which allows us to write the chi-square in terms of the variance of the conditional density:

[TABLE]

Inequality (d) follows from the non-negativity of the variance and the definition of $\kappa(t)$. ∎

Evaluating Proposition 7 with $t=1$ recovers the well-known inequality $I(X;Y)\leq\chi^{2}(P_{X,Y},P_{X}\times P_{Y})$. The next two results follow from the cases $0<t<1/2$ and $t=1/2$, respectively.

Proposition 8**.**

For any $0<r<1$, the mutual information satisfies

[TABLE]

where $t=(1-r)/(2-r)$.

Proof.

Starting with Proposition 7 and applying Hölder’s inequality with conjugate exponents $1/(1-t)$ and $1/t$ leads to

[TABLE]

where we have used the fact that $r=(1-2t)/(1-t)$. ∎

Proposition 9**.**

For any $p<1<q$, the mutual information satisfies

[TABLE]

where $\lambda=(q-1)/(q-p)$ and

[TABLE]

Proof.

Evaluating Proposition 7 with $t=1/2$ gives

[TABLE]

Evaluating Proposition 3 with $r=1/2$ leads to

[TABLE]

Combining these inequalities with the expression for $\psi_{1/2}(p,q)$ given in (2) completes the proof. ∎

The contribution of Propositions 8 and 9 is that they provide bounds on the mutual information in terms of quantities that can be easy to characterize. One application of these bounds is to establish conditions under which the mutual information corresponding to a sequence of random pairs $(X_{k},Y_{k})$ converges to zero. In this case, Proposition 8 provides a sufficient condition in terms of the Rényi entropy of $Y_{n}$ and the function $V_{0}(Y_{n}|X_{n})$, while Proposition 9 provides a sufficient condition in terms of $V_{s}(Y_{n}|X_{n})$ evaluated with two difference values of $s$. These conditions are summarized in the following result.

Proposition 10**.**

Let $(X_{k},Y_{k})$ be a sequence of random pairs such the conditional distribution $Y_{k}$ given $X_{k}$ has a density on $\mathbb{R}^{n}$. The following are sufficient conditions under which the mutual information of $I(X_{k};Y_{k})$ converges to zero as $k$ increases to infinity:

1. (i)

There exists $0<r<1$ such that

[TABLE] 2. (ii)

There exists $p<1<q$ such that

[TABLE]

IV-C Properties of the bounds

The function $V_{s}(Y|X)$ has a number of interesting properties. The variance of the conditional density can be expressed in terms of an expectation with respect to two independent random variables $X_{1}$ and $X_{2}$ with the same distribution as $X$ via the decomposition:

[TABLE]

Consequently, by swapping the order of the integration and expectation we obtain

[TABLE]

where

[TABLE]

The function $K_{s}(x_{1},x_{2})$ is a positive definite kernel that does not depend on the distribution of $X$. For $s=0$, this kernel has been studied previously in the machine learning literature [22], where it is referred to as the expected likelihood kernel.

The variance of the conditional density also satisfies a data-processing inequality. Suppose that $U\to X\to Y$ forms a Markov chain. Then, the square of the conditional density of $Y$ given $U$ can be expressed as

[TABLE]

where $(U,X^{\prime}_{1},X^{\prime}_{2})\sim P_{U}P_{X_{1}|U}P_{X_{2}|U}$. Combining this expression with (20) yields

[TABLE]

where we recall that $(X_{1},X_{2})$ are independent copies of $X.$

Finally, it is easy to verify that the function $V_{s}(Y)$ satisfies

[TABLE]

Using this scaling relationship we see that the sufficient conditions in Proposition 10 are invariant to scaling of $Y$.

IV-D Example with Gaussian noise

We now provide a specific example of our bounds on the mutual information. Let $(X,Y)$ be distributed according to

[TABLE]

were $W\sim\mathcal{N}(0,1)$ is independent of $X$. In this case, it is well known that the mutual information satisfies

[TABLE]

where equality is attained is $X$ is Gaussian. This inequality follows straightforwardly from the fact that the Gaussian distribution maximizes differential entropy subject to a second moment constraint. One of the limitations of this bound is that it can be loose when the second moment is dominated by events that have small probability. In fact, it is easy to construct examples for which $X$ does not have a finite second moment and yet $I(X;Y)$ is arbitrarily close to zero.

Our results provide bounds on $I(X;Y)$ that are significantly less sensitive to the effects of rare events. To begin, observe that the product of the conditional densities can be factored according to

[TABLE]

where $\phi(x)=(2\pi)^{-1/2}\exp(-x^{2}/2)$ is the density of the standard Gaussian distribution. Integrating with respect to $y$ leads to

[TABLE]

For the case $s=0$, the function $K_{0}(x_{1},x_{2})$ is proportional the standard Gaussian kernel and we have

[TABLE]

This expression shows that $V_{0}(Y|X)$ is a measure of the variation in $X$.

A useful property of $V_{0}(Y|X)$ is that the conditions under which it converges to zero are weaker than the conditions needed for other measures of variation, such that variance. To see why, observe that the expectation is bounded uniformly with respect to $(X_{1},X_{2})$. In particular, for every $\epsilon>0$ and $x\in\mathbb{R}$, we have

[TABLE]

where we have used the inequality $1-e^{-x}\leq x$ and the fact that \mathbb{P}\mathopen{}\mathclose{{}\left[|X_{1}-X_{2}|\geq 2\epsilon}\right]\leq 2\mathbb{P}\mathopen{}\mathclose{{}\left[|X-x|\geq\epsilon}\right]. Therefore, $V_{0}(Y|X)$ is small provided that $X$ is close a constant value with high probability.

To study some further properties of these bounds, we now focus on the case where $X$ is a Gaussian scalar mixture generated according to

[TABLE]

with $A$ and $U$ independent. In this case, the expectations with respect to the kernel $K_{s}(x_{1},x_{2})$ can be computed explicitly, leading to

[TABLE]

It can be shown that this expression depends primarily on the magnitude of $U$. This is not surprising given that $X$ converges to a constant if and only if $U$ converges to zero.

Our results can also be used to bound the mutual information $I(U;Y)$ by noting that $U\to X\to Y$ forms a Markov chain, and taking advantage of the characterization provided in (21). Letting $X_{1}^{\prime}=A_{1}\sqrt{U}$ and $X_{2}^{\prime}=A_{2}\sqrt{U}$ with $(A_{1},A_{2},U)$ mutually independent, leads to

[TABLE]

In this case, $V_{s}(Y|U)$ is a measure of the variation in $U$. To study it behavior, we consider the simple upper bound

[TABLE]

This bound shows that if $s\leq 1$ then $V_{s}(Y|U)$ is bounded uniformly with respect to distributions on $U$, and if $s>1$ then $V_{s}(Y|U)$ is bounded in terms of the $(\frac{s-1}{2})$-th moment of $U$.

In conjunction with Propositions 8 and 9 the function $V_{s}(Y|U)$ provide bounds on the mutual information $I(U;Y)$ that can be expressed in terms of simple expectations involving two independent copies of $U$. Figure 3 provides an illustration of the upper bound in Proposition 9 for the case where $U$ is a discrete random variable supported on two-points and $X$ and $Y$ are generated according to (22) and (24). This example shows that there exist sequences of distributions for which our upper bounds on the mutual information converges to zero while the chi-square divergence between $P_{XY}$ and $P_{X}\times P_{Y}$ is bounded away from zero.

V Conclusion

This paper provides bounds on Rényi entropy and mutual information that are based on a relatively simple two-moment inequality. One of the main takeaways from our analysis is that sometimes two carefully chosen moments are all that is needed to provide an accurate characterization. Extensions to inequalities with more moments are also worth exploring.

Appendix A The Gamma and Beta Functions

This section reviews some properties of the gamma and beta functions. For $x>0$, the gamma function is defined according to $\Gamma(x)=\int_{0}^{\infty}t^{x-1}e^{-t}\,\mathrm{d}t.$ Binet’s formula the logarithm of the gamma function [20, Theorem 1.6.3] gives

[TABLE]

where the remainder term $\theta(x)$ is convex and non-increasing with $\lim_{x\to 0}\theta(x)=\infty$ and $\lim_{x\to\infty}\theta(x)=0$. Euler’s reflection formula [20, Theorem 1.2.1] gives

[TABLE]

For $x,y>0$ the beta function is defined according to $\mathrm{B}(x,y)=\Gamma(x)\Gamma(y)/\Gamma(x+y)$. The beta function can also be expressed in integral form as [20, pg. 7]

[TABLE]

Recall that $\widetilde{\mathrm{B}}(x,y)=\mathrm{B}(x,y)(x+y)^{x+y}x^{-x}y^{-y}$. Using (25) leads to

[TABLE]

It can also be shown that [23, Equation (2) pg. 2]

[TABLE]

Appendix B Details for Rényi Entropy Examples

This appendix studies properties of the two-moment inequalities for Rényi entropy described in Section III.

B-A Lognormal distribution

Let $X$ be a lognormal random variable with parameters $(\mu,\sigma^{2})$ and consider the parametrization

[TABLE]

where $\lambda\in(0,1)$ and $u\in(0,\infty)$. Then, we have

[TABLE]

Combining these expressions with (28) leads to

[TABLE]

We now characterize the minimum with respect to the parameters $(\lambda,u)$. Note that the mapping $\lambda\mapsto\theta(\frac{r\lambda}{1-r})+\theta(\frac{r(1-\lambda)}{1-r})$ is convex and symmetric about the point $\lambda=1/2$. Therefore, the minimum with respect to $\lambda$ is attained at $\lambda=1/2$. Meanwhile, mapping $u\mapsto u\sigma^{2}-\log(u\sigma^{2})$ is convex and attains it minimum at $u=1/\sigma^{2}$. Evaluating (30) with these values, we see that the optimal two-moment inequality can be expressed as

[TABLE]

By (28), this expression is equivalent to (25). Moreover, the fact that $\Delta_{r}(X)$ decreases to zero as $r$ increases to one follows from the fact that $\theta(x)$ decreases to zero and $x$ increases to infinity.

Next, we express gap in terms of the pair $(p,q)$. Comparing the difference between $\Delta_{r}(X;p,q)$ and $\Delta_{r}(X)$ leads to

[TABLE]

where $\varphi(x)=x-\log(x)-1$. In particular, if $p=0$, then we obtain the simplified expression

[TABLE]

This characterization shows that the gap of the optimal one-moment inequality $\widetilde{\Delta}_{r}(X)$ increases to infinity in the limit as either $\sigma^{2}\to 0$ or $\sigma^{2}\to\infty$.

B-B Multivariate Gaussian distribution

Let $Y\sim\mathcal{N}(0,I_{n})$ is an $n$-dimensional Gaussian vector and consider the parametrization

[TABLE]

where $\lambda\in(0,1)$ and $z\in(0,\infty)$. The, we have

[TABLE]

Furthermore, if

[TABLE]

then $L_{r}(\|Y\|^{n};p,q)$ is finite and is given by

[TABLE]

where

[TABLE]

Combining these expressions and then using (25) and (28) leads to

[TABLE]

Next, we study some properties of $Q_{r,n}(\lambda,z)$. The decommissions (25) shows that the logarithm of the gamma function can expressed as the sum of convex functions:

[TABLE]

where $\varphi(x)=x\log x+1-x$. Starting with the definition of $Q(\lambda,z)$ and then using Jensen’s inequality yields

[TABLE]

where $a=2(1-r)/n$. Using the inequality $\varphi(x)\geq(3/2)(x-1)^{2}/(x+2)$ leads to

[TABLE]

where $b=2(1-r)/(9n)$.

Observe that the right-hand side of (34) converges to $z/2$ as $n$ increases to infinity. It turns out this limiting behavior is tight. Using (25), it is straightforward to show that $Q_{n}(\lambda,z)$ converges pointwise to $z/2$ as $n$ increases to infinity, that is

[TABLE]

for any fixed pair $(\lambda,z)\in(0,1)\times(0,\infty)$.

B-C Proof of Proposition 6

Let $D=(0,1)\times(0,\infty)$. For fixed $r\in(0,1)$ we use $Q_{n}(\lambda,z)$ to denote the function $Q_{r,n}(\lambda,z)$ defined in (32) and we use $G_{n}(\lambda,z)$ to denote the right-hand side of (33). These functions are defined to be equal to positive infinity for any pair $(\lambda,z)\in D$ such that (31) does not hold.

Note that the terms $\theta(n/(2r))$ and $\theta(n/2)$ converge to zero in the limit as $n$ increases to infinity. In conjunction with (35), this shows that $G_{n}(\lambda,z)$ converges pointwise to a limit $G(\lambda,z)$ given by

[TABLE]

At this point, the correspondence with the lognormal distribution can be seen from the fact that $G(\lambda,z)$ is equal to the right-hand side of (30) evaluated with $u\sigma^{2}=z$.

To show that the gap corresponding to the lognormal distribution provides an upper bound on the limit, we use

[TABLE]

Here, the last equality follows from the analysis in Appendix B-A, which shows that the minimum of $G(\lambda,z)$ is a attained at $\lambda=1/2$ and $z=1$.

To prove the lower bound requires a bit more work. Fix any $\epsilon\in(0,1)$ and let $D_{\epsilon}=(0,1-\epsilon]\times(0,\infty)$. Using the lower bound on $Q_{n}(\lambda,z)$ given in (34), it can be verified that

[TABLE]

Consequently, we have

[TABLE]

To complete the proof we will show that for any sequence $\lambda_{n}$ that converges to one as $n$ increases to infinity, we have

[TABLE]

To see why this is the case, note that by (28) and (29),

[TABLE]

Therefore, we can write

[TABLE]

where $c_{n}$ is bounded uniformly for all $n$. Making the substitution $u=\lambda(1-\lambda)z$, we obtain

[TABLE]

Next, let $b_{n}=2(1-r)/(9n)$. The lower bound in (34) leads to

[TABLE]

The limiting behavior in (38) can now be seen as a consequence of (39) and the fact that, for any sequence $\lambda_{n}$ converging to one, the right-hand side (40) increases without bound as $n$ increases. Combining (36), (37), and (38) establishes that the large $n$ limit of $\Delta_{r}(Y)$ exists and is equal to $\Delta_{r}(X)$. This concludes the proof of Proposition 6

B-D Proof of Inequality (17)

Given any $\lambda\in(0,1)$ and $u\in(0,\infty)$ let

[TABLE]

We need the following results, which characterize the terms in Proposition 4 in the limit as $r$ increases to one.

Lemma 11**.**

The function $\psi_{r}(p(r),q(r))$ satisfies

[TABLE]

Proof.

Starting with (28), we can write

[TABLE]

As $r$ converges to one the terms in the exponent converge to zero. Noting that $q(r)-p(r)=\sqrt{r\lambda(1-\lambda)/(1-r)}$ completes the proof. ∎

Lemma 12**.**

If $X$ is a random variable such that s\mapsto\mathbb{E}\mathopen{}\mathclose{{}\left[|X\|^{s}}\right] is finite in a neighborhood of zero, then \mathbb{E}\mathopen{}\mathclose{{}\left[\log(X)}\right] and $\operatorname{\sf Var}(\log(X))$ are finite, and

[TABLE]

Proof.

Let \Lambda(s)=\log(\mathbb{E}\mathopen{}\mathclose{{}\left[|X|^{s}}\right]). The assumption that \mathbb{E}\mathopen{}\mathclose{{}\left[|X|^{s}}\right] is finite in a neighborhood of zero means that \mathbb{E}\mathopen{}\mathclose{{}\left[(\log|X|)^{m}}\right] is finite for all positive integers $m$ that $\Lambda(s)$ is real-analytic in a neighborhood of zero, that is there exists constants $\delta>0$ and $C<\infty$, depending on $X$, such that

[TABLE]

where a=\mathbb{E}\mathopen{}\mathclose{{}\left[\log|X|}\right] and $b=\frac{1}{2}\operatorname{\sf Var}(|X|)$. Consequently, for all $r$ such that $1-\delta<p(r)<(1-r)/r<q(r)<1+\delta$, it follows that

[TABLE]

Taking the limit as $r$ increases to one completes the proof. ∎

We are now ready to prove Inequality (17). Combining Proposition 4 with Lemma 11 and Lemma 12 yields

[TABLE]

The stated inequality follows from evaluating the right-hand side with $u=1/\operatorname{\sf Var}(\log X)$ and recalling that $h(X)$ corresponds to the limit of $h_{r}(X)$ as $r$ increases to one.

Appendix C Properties of Logarithm-Power Ratio

This section studies properties of the function $\kappa:(0,1]\to\mathbb{R}_{+}$ defined by

[TABLE]

For $t=1$, the bound $\log(1+u)\leq u$ means that $\kappa(1)\leq 1$. Noting $\lim_{u\to 0}\log(1+u)/u=1$ shows that this inequality is tight, and thus $\kappa(1)=1$. For any $t\in(0,1)$, it can be verified via differentiation that the supremum is attained on $(0,\infty)$ by the unique solution $u^{*}_{t}$ to the fixed-point equation

[TABLE]

The solution to this equation can be expressed as

[TABLE]

where Lambert’s function $W(z)$ is the solution to the equation $z=x\exp(x)$ on the interval on $[-1,\infty)$.

Lemma 13**.**

The function $g(t)=t\kappa(t)$ is nondecreasing on $(0,1]$ with $\lim_{t\to 0}g(t)=1/e$ and $g(1)=1$.

Proof.

The fact that $g(1)=1$ follows from $\kappa(1)=1$. By the envelope theorem [24], the derivative of $g(t)$ can be expressed as

[TABLE]

Therefore, the derivative satisfies

[TABLE]

Noting that $u\mapsto u\log u$ is negative on $(0,1)$ and nonnegative and nondecreasing on $[1,\infty)$ shows that the last condition is always satisfied, and hence $g^{\prime}(t)$ is nonnegative.

To prove the small $t$ limit we can rearrange (42) to see that $u^{*}_{t}$ satisfies

[TABLE]

and hence

[TABLE]

Now, as $t$ decreases to zero, (43) shows that $u_{t}^{*}$ increases to infinity. By (44), it then follows that $\log(g(t))$ converges to negative one, which proves the desired limit. ∎