Comment on the Equality Condition for the I-MMSE Proof of Entropy Power Inequality

TL;DR

This paper clarifies the conditions under which the I-MMSE proof of the entropy power inequality achieves equality, by deriving an exact expression for the deficit and linking it to the Cauchy functional equation.

Contribution

It provides the first precise characterization of the equality condition in the I-MMSE proof of the EPI, connecting it to the Cauchy functional equation.

Findings

Derived an exact expression for the EPI deficit.

Identified a necessary condition for equality involving the Cauchy functional equation.

Enhanced understanding of the equality case in the I-MMSE proof.

Abstract

The paper establishes the equality condition in the I-MMSE proof of the entropy power inequality (EPI). This is done by establishing an exact expression for the deficit between the two sides of the EPI. Interestingly, a necessary condition for the equality is established by making a connection to the famous Cauchy functional equation.

Equations

$${\mathrm{e}}^{\frac{2}{n}h(\mathbf{V}+\mathbf{W})}\ge {\mathrm{e}}^{\frac{2}{n}h(\mathbf{V})}+{\mathrm{e}}^{\frac{2}{n}h(\mathbf{W})},$$

$${\mathbf{X}}_1$$

$$\alpha$$

$$h\left(\sqrt{1-\alpha }{\mathbf{X}}_1+\sqrt{\alpha }{\mathbf{X}}_2\right)\ge \alpha h({\mathbf{X}}_1)+(1-\alpha )h({\mathbf{X}}_2),\forall \alpha \in [0,1],$$

$$\mathbb{E}\left[\Vert \mathbf{X}-\mathbb{E}[\mathbf{X}\mid \mathbf{Y}]{\Vert }^2\right]=\mathbb{E}\left[\Vert \mathbf{X}-f(\mathbf{Y}){\Vert }^2\right]-\mathbb{E}\left[{\Vert f(\mathbf{Y})-\mathbb{E}[\mathbf{X}\mid \mathbf{Y}]\Vert }^2\right].$$

$$\mathbb{E}\left[{\Vert f(\mathbf{Y})-\mathbb{E}\left[\mathbf{X}\mid \mathbf{Y}\right]\Vert }^2\right]$$

$$=\mathbb{E}\left[\Vert \mathbf{X}-f(\mathbf{Y}){\Vert }^2\right]+\mathbb{E}\left[\Vert \mathbf{X}-\mathbb{E}\left[\mathbf{X}\mid \mathbf{Y}\right]{\Vert }^2\right]+2\mathbb{E}\left[\mathrm{Tr}\{(f(\mathbf{Y})-\mathbf{X})(\mathbf{X}-\mathbb{E}\left[\mathbf{X}\mid \mathbf{Y}\right]{)}^T\}\right]$$

$$=\mathbb{E}\left[\Vert \mathbf{X}-f(\mathbf{Y}){\Vert }^2\right]+\mathbb{E}\left[(\mathbf{X}-\mathbb{E}\left[\mathbf{X}\mid \mathbf{Y}\right]{)}^2\right]-2\mathbb{E}\left[\mathrm{Tr}\{\mathbf{X}(\mathbf{X}-\mathbb{E}\left[\mathbf{X}\mid \mathbf{Y}\right]{)}^T\}\right]$$

$$=\mathbb{E}\left[\Vert \mathbf{X}-f(\mathbf{Y}){\Vert }^2\right]+\mathbb{E}\left[\Vert \mathbf{X}-\mathbb{E}\left[\mathbf{X}\mid \mathbf{Y}\right]{\Vert }^2\right]-2\mathbb{E}\left[\Vert \mathbf{X}-\mathbb{E}\left[\mathbf{X}\mid \mathbf{Y}\right]{\Vert }^2\right]$$

$$=\mathbb{E}\left[\Vert \mathbf{X}-f(\mathbf{Y}){\Vert }^2\right]-\mathbb{E}\left[\Vert \mathbf{X}-\mathbb{E}\left[\mathbf{X}\mid \mathbf{Y}\right]{\Vert }^2\right],$$

$$f(\mathbf{x}+\mathbf{y})=f(\mathbf{x})+f(\mathbf{y}),$$

$$\mathbb{E}\left[a_1{f}_1(\mathbf{V})+a_2{f}_2(\mathbf{W})\mid a_1\mathbf{V}+a_2\mathbf{W}\right]=a_1{f}_1(\mathbf{V})+a_2{f}_2(\mathbf{W})\text{ a.s., }$$

$$f_1(\mathbf{v})=\mathbf{A}\mathbf{v}+\mathbf{b},f_2(\mathbf{w})=\mathbf{A}\mathbf{w}+\mathbf{c},$$

$$h(a_1\mathbf{W}+a_2\mathbf{V})=a_1{f}_1(\mathbf{V})+a_2{f}_2(\mathbf{W}).$$

$$h(a_1\mathbf{v}+a_2\mathbf{w})=a_1{f}_1(\mathbf{v})+a_2{f}_2(\mathbf{w}),$$

$$h(a_1\mathbf{v})=a_1{f}_1(\mathbf{v})+a_2{f}_2(0),$$

$$h(a_2\mathbf{w})=a_1{f}_1(0)+a_2{f}_2(\mathbf{w}).$$

$$h(a_1\mathbf{v})+h(a_2\mathbf{w})$$

$$h(\mathbf{x})=\mathbf{A}\mathbf{x}+\mathbf{a},$$

$$\mathbb{E}\left[\mathbb{E}\left[\mathbf{X}\mid {\mathcal{G}}_2\right]\mid {\mathcal{G}}_1\right]=\mathbb{E}\left[\mathbf{X}\mid {\mathcal{G}}_1\right]\text{ a.s.}$$

$$\mathbb{E}\left[\mathbb{E}\left[\mathbf{X}\mid \mathbf{W},\mathbf{V}\right]\mid \mathbf{W}+\mathbf{V}\right]=\mathbb{E}\left[\mathbf{X}\mid \mathbf{W}+\mathbf{V}\right]\text{ a.s.}$$

$$\mathbb{E}[\mathbf{X}\mid \mathbf{Y}]=\mathbf{A}\mathbf{Y}+\mathbf{v}\text{ a.s.},$$

$$\mathbf{A}={\mathbf{K}}_{\mathbf{X}\mathbf{Y}}{\mathbf{K}}_{\mathbf{Y}}^{-1}={\mathbf{K}}_{\mathbf{X}}(\mathbf{I}+{\mathbf{K}}_{\mathbf{X}}{)}^{-1},\mathbf{v}=(\mathbf{I}-\mathbf{A}){\mu }_{\mathbf{X}}.$$

$$0$$

$$=\mathbb{E}\left[\left((1-a)X-Z\right){\mathrm{e}}^{-itY}\right]$$

$$=(1-a)\mathbb{E}\left[X{\mathrm{e}}^{-itY}\right]-\mathbb{E}\left[Z{\mathrm{e}}^{-itY}\right]$$

$$=(1-a)\mathbb{E}\left[X{\mathrm{e}}^{-itX}\right]\mathbb{E}\left[{\mathrm{e}}^{-itZ}\right]-\mathbb{E}\left[Z{\mathrm{e}}^{-itZ}\right]\mathbb{E}\left[{\mathrm{e}}^{-itX}\right]$$

$$=(1-a)i{\varphi }_X^{{}^{\prime }}(t){\varphi }_Z(t)-i{\varphi }_Z^{{}^{\prime }}(t){\varphi }_X(t)$$

$$=(1-a)i{\varphi }_X^{{}^{\prime }}(t){\mathrm{e}}^{-\frac{t^2}{2}}-it{\mathrm{e}}^{-\frac{t^2}{2}}{\varphi }_X(t),$$

$$(1-a){\varphi }_X^{{}^{\prime }}(t)=t{\varphi }_X(t).$$

$${\rho }_{\mathbf{X}}(\mathbf{x})$$

$$J(\mathbf{X})$$

$${\rho }_{\mathbf{Y}}(\mathbf{Y})=\sqrt{\gamma }\mathbb{E}[\mathbf{X}\mid \mathbf{Y}]-\mathbf{Y}\text{ a.s.}$$

$$h(\mathbf{X})=\frac{1}{2}{\int }_0^{\infty }\mathbb{E}\left[\Vert \mathbf{X}-\mathbb{E}[\mathbf{X}\mid {\mathbf{Y}}_{\gamma }]{\Vert }^2\right]-\frac{n}{2\pi \mathrm{e}+\gamma }d\gamma ,$$

$$\underset{t\to 0}{\lim }h(\mathbf{X}+t\mathbf{Z})=h(\mathbf{X}),$$

$$h(\sqrt{1-\alpha }{\mathbf{X}}_1+\sqrt{\alpha }{\mathbf{X}}_2)=(1-\alpha )h({\mathbf{X}}_1)+\alpha h({\mathbf{X}}_2)+\Delta ({\mathbf{X}}_1\Vert {\mathbf{X}}_2),$$

$$\Delta ({\mathbf{X}}_1\Vert {\mathbf{X}}_2)$$

$$\mathbf{X}$$

$${\mathbf{Y}}_{1,\gamma }$$

$${\mathbf{Y}}_{2,\gamma }$$

Full text

The classical entropy power inequality (EPI) formulated by Shannon in [1] states that for two independent continuous random vectors ${\bf V}$ and ${\bf W}$

[TABLE]

where equality in (1) is attained if and only if ${\bf V}$ and ${\bf W}$ are Gaussian with proportional covariances (i.e., ${\bf K}_{W}=c\,{\bf K}_{V}$ for some scalar $c>0$). Via the transformation

[TABLE]

the EPI can be shown to be equivalent to Lieb’s inequality [2]

[TABLE]

and where equality in (2) holds if and only if ${\bf K}_{{\bf X}_{1}}={\bf K}_{{\bf X}_{2}}$.

There are several proofs of the EPI which follow three distinct methods: using integration over a path of a continuous Gaussian perturbation [3, 4, 5, 6, 7]; using the sharp version of Young’s inequality and properties of Rényi entropy [2, 5, 4]; and using a change of variable and Knöthe’s map [8, 9]. For a comprehensive list of references and a detailed history of the EPI, the reader is referred to [10] and references therein.

As was recently pointed out in [8] not all available proofs settle the equality case in (1) and (2). In particular, for the class of proofs via Gaussian perturbations, the case of equality has not yet been established in the proof given in [6], which relies on the so-called I-MMSE relationship [11].

The goal of this paper is to close this gap by establishing the equality case in the proof of the EPI via the I-MMSE relationship. Equality is established by determining an exact expression for the deficit in (2) and showing that the deficit is zero if and only if ${\bf X}_{1}$ and ${\bf X}_{2}$ are Gaussian with identical covariances.

Notation

Deterministic scalar/vector quantities are denoted by lowercase normal/bold letters, matrices by bold uppercase letters, random variables by uppercase letters, and random vectors by bold uppercase letters. For a random vector ${\bf V}$ we denote the covariance matrix by ${\bf K}_{{\bf V}}$, determinant by $|{\bf K}_{{\bf V}}|$, transpose by ${\bf V}^{T}$, and trace by ${\rm Tr}\{{\bf V}\}$. The Euclidian norm of a vector ${\bf v}$ is denoted by $\|{\bf v}\|$. The gradient operator is denoted by $\nabla$. The $\mathbb{E}[\cdot]$ denotes the expectation operator.

Assumptions

Throughout the paper, we assume that all random vectors treated in this work have covariance matrices with bounded entries and continuous, positive, and differentiable probability densities. Therefore, quantities such as entropies, expectations, and conditional expectations are well defined throughout the paper. The interested reader is referred to [8] and [12] where it is shown that the set of aforementioned assumptions is sufficient to prove the EPI in (1).

I Preliminary Results

In this section, we present necessary mathematical tools needed in this paper.

The first result of this section establishes the penalty, incurred in the minimum mean square error (MMSE), for using a sub-optimal estimator.

Lemma 1**.**

Let $f:\mathbb{R}^{n}\to\mathbb{R}^{n}$ be measurable and such that $\mathbb{E}\left[\|f({\bf Y})\|^{2}\right]\leq\infty$. Then,

[TABLE]

Proof:

[TABLE]

where (4a) and (4b) are due to the orthogonality principle. This concludes the proof. ∎

The necessary condition for the equality in (2) will be shown to be a consequence of a remarkably simple, yet powerful, Cauchy functional equation.

Lemma 2**.**

(Cauchy Functional Equation.) Over a space of measurable111In this paper, measurable is meant with respect to the Lebesgue measure. functions from $\mathbb{R}^{n}$ to $\mathbb{R}^{n}$ the equation

[TABLE]

is satisfied if and only if $f({\bf x})={\bf A}\,{\bf x}$ (i.e., is linear) for some matrix ${\bf A}\in\mathbb{R}^{n\times n}$.

Proof:

See [13, Chapter 2]. ∎

Cauchy functional equation has a very rich history, and the interested reader is referred to [13] for a comprehensive summary. Cauchy functional equation is used next to establish the following property of the conditional expectation.

Lemma 3**.**

Let ${\bf V},{\bf W}\in\mathbb{R}^{n}$ be independent random vectors with support and $f_{1},\,f_{2}:\mathbb{R}^{n}\to\mathbb{R}^{n}$ be measurable functions such that $\mathbb{E}\left[\|f_{1}({\bf V})\|^{2}\right],\,\mathbb{E}\left[\|f_{2}({\bf W})\|^{2}\right]<\infty$. Then, for any $a_{1},a_{2}\in\mathbb{R}$

[TABLE]

for some ${\bf A}\in\mathbb{R}^{n\times n}$ and ${\bf b},{\bf c}\in\mathbb{R}^{n}$.

Proof:

The proof of the sufficient condition follows trivially. To show the necessary condition observe that (6a) is equivalent to identifying a set of functions $\{h(\cdot)\}$ for which

[TABLE]

Since ${\bf V}$ and ${\bf W}$ are fully supported, we have that

[TABLE]

for all $({\bf v},{\bf w})$. In particular,

[TABLE]

Therefore, by adding the two equations in (9) and using (8), we arrive at

[TABLE]

Next, by letting $f({\bf x})=h({\bf x})-h(0)$, it is not difficult to see that (10) corresponds to Cauchy functional equation in Lemma 2. As a result, we concluded that $h(\cdot)$ is an affine function

[TABLE]

for some ${\bf A}\in\mathbb{R}^{n\times n}$ and ${\bf a}\in\mathbb{R}^{n}$. Finally, (11) and (9) imply that functions $f_{1}(\cdot)$ and $f_{2}(\cdot)$ are also affine with the same slope. This concludes the proof. ∎

The following well-known property of the conditional expectation will be useful in manipulating some of our expressions.

Lemma 4**.**

(Smoothing or Towering Property of the Conditional Expectation.) Let sigma algebras $\mathcal{G}_{1},\mathcal{G}_{2}$ be such that $\mathcal{G}_{1}\subset\mathcal{G}_{2}$. Then,

[TABLE]

Proof:

See [14, Chapter 10]. ∎

The key step of the proof would be to establish that the equality holds if and only if a certain conditional expectation is a linear or an affine function. The following result shows that the conditional expectation is an affine function if and only if the input random variable is Gaussian.

Lemma 5**.**

Let ${\bf Y}=\,{\bf X}+{\bf Z}$ where ${\bf X}$ and ${\bf Z}\sim\mathcal{N}(0,{\bf I})$ are independent. Then,

[TABLE]

Proof:

Lemma 5 is a well known result from estimation theory and the details of the proof can be found in [15]. Here, we only a give sketch of the proof of the necessary condition. To show the necessary condition, one must show that, in the MMSE sense, linear estimators are only optimal for Gaussian random vectors. For simplicity, we only look at the zero mean (i.e., $\mu_{X}=\mathbb{E}[X]=0$ and $v=0$) and the scalar case. Let $f(Y)={\rm e}^{-itY}$, and let $aY$ be an estimator that we claim to be optimal with $a=\frac{E[XY]}{E[Y^{2}]}$. Then, by the orthogonality principle we have that

[TABLE]

where (14a) follows by the independence of $X$ and $Z$, and (14b) follows by the derivative expression $\phi_{X}^{{}^{\prime}}(t)=-i\,\mathbb{E}\left[X{\rm e}^{-itX}\right]$ (the derivative expression holds since by assumption $\mathbb{E}[X^{2}]<\infty)$.

Therefore, from (14c) we have a differential equation of the form

[TABLE]

The only nontrivial solution to the differential equation in (15) is given by the Gaussian distribution with the characteristic function given by $\phi_{X}(t)={\rm e}^{-(a-1)\frac{t^{2}}{2}}$. This concludes the proof. ∎

We define the score function and the Fisher information of a continuous random vector ${\bf X}$ with the probability density function $f_{{\bf X}}({\bf x})$ as

[TABLE]

For the Gaussian noise channel, the score function of the output can be related to the conditional expectation.

Lemma 6**.**

Let ${\bf Y}=\sqrt{\gamma}\,{\bf X}+{\bf Z}$ where ${\bf X}$ and ${\bf Z}\sim\mathcal{N}(0,{\bf I})$ are independent. Then,

[TABLE]

Proof:

See [11, Eq.(56)]. ∎

We conclude this section by giving an expression for the differential entropy in terms of an integral of the MMSE which is a consequence of the I-MMSE relationship in [11].

Lemma 7**.**

For every continuous random vector ${\bf X}\in\mathbb{R}^{n}$,

[TABLE]

where ${\bf Y}_{\gamma}=\sqrt{\gamma}\,{\bf X}+{\bf Z}$ and ${\bf X}$ is independent of ${\bf Z}\sim\mathcal{N}(0,{\bf I})$.

II Main Results

The first main result of this section, which is a refinement of the bound in [6], establishes an exact expression for the deficit in (2).

Theorem 1**.**

For any independent continuous random vectors ${\bf X}_{1},{\bf X}_{2}\in\mathbb{R}^{n}$ and any $\alpha\in[0,1]$

[TABLE]

Proof:

According to Lemma 7, the entropy of a random vector ${\bf X}\in\mathbb{R}^{n}$, defined in (19c), is given by

[TABLE]

where the last step follows by taking

[TABLE]

in Lemma 1.

Next, by the mutual independence of $({\bf X}_{1},{\bf X}_{2},{\bf Z}_{1},{\bf Z}_{2})$, the first expectation in (20) reduces to

[TABLE]

Finally, by combining (20) and (22) we arrive at

[TABLE]

where $\Delta({\bf X}_{1}\|{\bf X}_{2})$ is defined in (19b). This concludes the proof. ∎

Clearly, $\Delta({\bf X}_{1}\|{\bf X}_{2})$ in (19b) is a non-negative quantity which leads to Lieb’s inequality in (2).

II-A On the Equality Condition

The following result establishes necessary and sufficient conditions for the equality in (2) and gives several equivalent statemens for the equality.

Theorem 2**.**

The following statements are equivalent:

[TABLE]

Moreover, equality in (23) holds if and only if ${\bf X}_{1}$ and ${\bf X}_{2}$ are Gaussian with identical covariances.

Proof:

From (19b) it is immediate that $\Delta({\bf X}_{1}\|{\bf X}_{2})=0$ if and only if

[TABLE]

This shows equivalence between (23a) and (23b).

The equivalence between (23b) and (23c) follows from the towering property in Lemma 4

[TABLE]

Showing equivalence between (23d), (23e) and (23f) is deferred to Appendix A.

Next, we show that (23) is satisfied if and only if ${\bf X}_{1}$ and ${\bf X}_{2}$ are Gaussian random vectors with identical covariances. The sufficient condition follows by noting that if ${\bf X}_{1}\sim\mathcal{N}(0,{\bf K}_{{\bf X}_{1}})$ and ${\bf X}_{2}\sim\mathcal{N}(0,{\bf K}_{{\bf X}_{2}})$, then the estimators are linear and are given by

[TABLE]

Therefore, the equality condition in (23b) holds only if

[TABLE]

With a small amount of algebra it is not difficult to show that the equality in (26) holds only if ${\bf K}_{{\bf X}_{1}}={\bf K}_{{\bf X}_{2}}$.

The necessary condition follows by letting $f_{1}({\bf Y}_{1,\gamma})=\,\mathbb{E}[{\bf X}_{1}\mid{\bf Y}_{1,\gamma}]$ and $f_{2}({\bf Y}_{2,\gamma})=\mathbb{E}[{\bf X}_{2}\mid{\bf Y}_{2,\gamma}]$, in which case the condition in (23c) reduces to

[TABLE]

According to Lemma 3 equality in (27) implies that $f_{1}({\bf Y}_{1,\gamma})$ and $f_{2}({\bf Y}_{2,\gamma})$ (or $\mathbb{E}[{\bf X}_{1}\mid{\bf Y}_{1,\gamma}]$ and $\mathbb{E}[{\bf X}_{2}\mid{\bf Y}_{2,\gamma}]$) are affine functions with the same slope. In other words, the conditional expectations are given by

[TABLE]

Moreover, by Lemma 5 the linearity of conditional expectations implies that ${\bf X}_{1}$ and ${\bf X}_{2}$ are Gaussian random vectors such that

[TABLE]

From (28), it is evident that ${\bf X}_{1}$ and ${\bf X}_{2}$ have identical covariances. This concludes the proof. ∎

III Concluding Remark

In this work, we have established the equality condition for the I-MMSE proof of the EPI. Theorem 2 also establishes an equality condition for the following Fisher information inequality

[TABLE]

This should come as no surprise since the inequality in (29) is a key to establishing the proof of the EPI via DeBruijn’s identity [3, 4, 5]. The equality condition in (29) was previously established in [16] by showing that a certain differential equation is satisfied only by Gaussian densities, and in [17] by checking the equality case of the Variance Drop inequality. In contrast, our proof relies on Cauchy functional equation and towering property of the conditional expectation.

It is also interesting to observe that the expression for the deficit

[TABLE]

is closely related to the mismatched representation of the relative entropy [18]

[TABLE]

where ${\bf X}_{1}\sim P,\,{\bf X}_{2}\sim Q$, and $\sqrt{\gamma}\,{\bf X}_{1}+{\bf Z}\sim P_{{\bf Y}_{\gamma}},\,\sqrt{\gamma}\,{\bf X}_{2}+{\bf Z}\sim Q_{{\bf Y}_{\gamma}}$, and where

[TABLE]

is the relative Fisher information distance.

Moreover, in view of the exact characterization of $\Delta({\bf X}_{1}\|{\bf X}_{2})$ in (19b), it would be interesting to explore connections to the work in [19]. The authors of [19] provided lower bounds on $\Delta({\bf X}_{1}\|{\bf X}_{2})$, for log-concave densities in terms of Wasserstein distance, by using the lower bound on $\Delta({\bf X}_{1}\|{\bf X}_{2})$ from [8].

Appendix A Proof of equivalence between (23d), (23e), and (23f)

The equivalence between (23c) and (23d) follows from Lemma 6.

Next we show the equivalence between (23d) and (23e). Using Lemma 6 the score function can be written as

[TABLE]

Next, by the towering property of the conditional expectation in Lemma 4

[TABLE]

where the last step follows Lemma 6. Putting equation (35) and (36) together we arrive at

[TABLE]

which establishes equivalence between (23d) and (23e). The expression in (37) is sometimes called a convolution identity of the score function [17].

The equivalence between (23d) and (23f) follows from (37) and the definition of Fisher’s information in (16). This concludes the proof.