Tight Bounds on the Weighted Sum of MMSEs with Applications in Distributed Estimation

TL;DR

This paper derives tight bounds on the weighted sum of MMSEs in Gaussian channels using KL divergence constraints, providing insights for robust distributed estimation in wireless systems.

Contribution

It introduces new tight bounds on MMSE sums constrained by KL divergence, characterizes the optimal Gaussian distributions, and demonstrates robustness of estimators against distribution deviations.

Findings

Derived tight upper and lower bounds on weighted MMSE sums.

Identified Gaussian distributions that attain these bounds.

Showed estimators that attain the upper bound are minimax robust.

Abstract

In this paper, tight upper and lower bounds are derived on the weighted sum of minimum mean-squared errors for additive Gaussian noise channels. The bounds are obtained by constraining the input distribution to be close to a Gaussian reference distribution in terms of the Kullback--Leibler divergence. The distributions that attain these bounds are shown to be Gaussian whose covariance matrices are defined implicitly via systems of matrix equations. Furthermore, the estimators that attain the upper bound are shown to be minimax robust against deviations from the assumed input distribution. The lower bound provides a potentially tighter alternative to well-known inequalities such as the Cram\'{e}r--Rao lower bound. Numerical examples are provided to verify the theoretical findings of the paper. The results derived in this paper can be used to obtain performance bounds, robustness guarantees, and engineering guidelines for the design of local estimators for distributed estimation problems which commonly arise in wireless communication systems and sensor networks.

Equations

$$\operatorname{mse}_{X|Y_{j}}(f_{j},P_{X})\coloneqq E_{P_{Y_{j}|X}P_{X}}\Bigl{[}\left\lVert f_{j}(Y_{j})-X\right\rVert^{2}\Bigr{]}.$$

$${\mathrm{mmse}}_{X|Y_j}(P_X):=\underset{f_j\in \mathcal{F}}{\inf }{\mathrm{mse}}_{X|Y_j}(f_j,P_X),$$

$$\mathcal{F}=\left\{f:({\mathbb{R}}^K,{\mathcal{B}}^K)\to ({\mathbb{R}}^K,{\mathcal{B}}^K)\right\}.$$

$$\underset{P_X\in {\mathcal{P}}_{\epsilon }}{\sup }\sum _{j=1}^J{\lambda }_j{\mathrm{mmse}}_{X|Y_j}(P_X)$$

$$\underset{P_X\in {\mathcal{P}}_{\epsilon }}{\inf }\sum _{j=1}^J{\lambda }_j{\mathrm{mmse}}_{X|Y_j}(P_X)$$

$$\mathcal{P}_{\varepsilon}\coloneqq\bigl{\{}P_{X}:D_{\text{KL}}(P_{X}\,\|\,P_{0})\leq\varepsilon\bigr{\}}.$$

$$P_0=\mathcal{N}({\mu }_0,{\Sigma }_0).$$

$$W_j$$

$${\mathrm{MMSE}}_j$$

$${\mathrm{SNR}}_0$$

$${\Sigma }_X={\Sigma }_0+\alpha \left(\sum _{j=1}^J{\lambda }_j{\mathrm{MMSE}}_j^{\text{T}}{\mathrm{MMSE}}_j\right){\mathrm{SNR}}_0^{-1}$$

$$\displaystyle\operatorname{tr}\bigl{(}\operatorname{SNR}_{0}\bigr{)}-\log\det\bigl{(}\operatorname{SNR}_{0}\bigr{)}-K=2\varepsilon,$$

$$P_{X}^{*}=\mathcal{N}\bigl{(}\mu_{0},\Sigma_{X}^{*}\bigr{)}$$

$$P_{X}^{\dagger}=\mathcal{N}\bigl{(}\mu_{0},\Sigma_{X}^{\dagger}\bigr{)}$$

$$f_j(y_j)=(I-W_j)y_j+W_j{\mu }_0,$$

$${\mathrm{mmse}}_{X|Y_j}(P_X)$$

$$\displaystyle=\operatorname{tr}\bigl{(}\operatorname{MMSE}_{j}\bigr{)}.$$

$$\displaystyle\sum_{j=1}^{J}\lambda_{j}\operatorname{mmse}_{X|Y_{j}}(P_{X})\leq\sum_{j=1}^{J}\lambda_{j}\operatorname{tr}\Bigl{(}\operatorname{MMSE}_{j}^{*}\Bigr{)}$$

$$\displaystyle\sum_{j=1}^{J}\lambda_{j}\operatorname{mmse}_{X|Y_{j}}(P_{X})\geq\sum_{j=1}^{J}\lambda_{j}\operatorname{tr}\Bigl{(}\operatorname{MMSE}_{j}^{\dagger}\Bigr{)},$$

$$\underset{P_X\in {\mathcal{P}}_{\epsilon }}{\sup }\underset{f_j\in \mathcal{F}}{\inf }\sum _{j=1}^J{\lambda }_j{\mathrm{mse}}_{X|Y_j}(f_j,P_X),$$

$$\sum _{j=1}^J{\lambda }_j{\mathrm{mse}}_{X|Y_j}(f_j^{\ast },P_X^{\ast })\le \sum _{j=1}^J{\lambda }_j{\mathrm{mse}}_{X|Y_j}(f_j,P_X^{\ast })$$

$$\sum _{j=1}^J{\lambda }_j{\mathrm{mse}}_{X|Y_j}(f_j^{\ast },P_X^{\ast })\ge \sum _{j=1}^J{\lambda }_j{\mathrm{mse}}_{X|Y_j}(f_j^{\ast },P_X)$$

$$\sum_{j=1}^{J}\lambda_{j}\operatorname{mse}_{X|Y_{j}}(f_{j}^{*},P_{X})=E_{P_{X}}\bigl{[}h(X)\bigr{]},$$

$$h(x)$$

$$\displaystyle=\sum_{j=1}^{J}\lambda_{j}E_{\mathcal{N}(x,\Sigma_{N_{j}})}\Bigl{[}\left\lVert(I-W_{j})Y_{j}+W_{j}\mu_{0}-x\right\rVert_{2}^{2}\Bigr{]}$$

$$=c+\sum _{j=1}^J{\lambda }_j{\Vert (I-W_j)x+W_j{\mu }_0-x\Vert }_2^2$$

$$=c+\sum _{j=1}^J{\lambda }_j(x-{\mu }_0{)}^{\text{T}}{W}_j^{\text{T}}{W}_j(x-{\mu }_0)$$

$$=c+(x-{\mu }_0{)}^{\text{T}}\left(\sum _{j=1}^J{\lambda }_j{W}_j^{\text{T}}{W}_j\right)(x-{\mu }_0),$$

$$p_X^{\ast }(x)=p_0(x)e^{\alpha h(x)+\beta -1},$$

$$p_X^{\ast }(x)$$

$$\displaystyle\propto e^{-\frac{1}{2}(x-\mu_{0})^{\text{T}}\bigl{(}\Sigma_{0}^{-1}-\alpha\sum_{j=1}^{J}\lambda_{j}W_{j}^{\text{T}}W_{j}\bigr{)}(x-\mu_{0})}$$

$$=e^{-\frac{1}{2}(x-{\mu }_0{)}^{\text{T}}{\Sigma }_X^{-1}(x-{\mu }_0)},$$

$${\Sigma }_X^{-1}={\Sigma }_0^{-1}-\alpha \sum _{j=1}^J{\lambda }_j{W}_j^{\text{T}}{W}_j.$$

$$D_{\text{KL}}(P_{X}\,\|\,P_{0})=\frac{1}{2}\left(\operatorname{tr}\bigl{(}\operatorname{SNR}_{0}\bigr{)}-K-\log\det\bigl{(}\operatorname{SNR}_{0}\bigr{)}\right).$$

$$\underset{Q}{\min }{D}_{\text{KL}}(P_X\Vert Q)<\infty .$$

$${\mathrm{mmse}}_{X|Y_j}(P_X)$$

Full text

I Introduction

The mean square error (MSE) is a natural and commonly used measure for the accuracy of an estimator. The minimum MSE (MMSE) plays a central role in statistics [1, 2], information theory [3, 4], signal processing [5, 6, 7], and has close connections to entropy and mutual information [8, 9]. In [10] and [11], lower and upper bounds on the MMSE are derived when the random variable of interest is contaminated by additive Gaussian noise and its distribution is $\varepsilon$-close to a Gaussian reference distribution in terms of the Kullback–Leibler (KL) divergence. The estimator that attains this upper bound is shown to be minimax robust in the sense that it minimizes the maximum MMSE over the set of feasible distributions. That is, within the specified KL divergence ball, it is robust to arbitrary deviations of the prior from the nominal Gaussian case. The lower bound provides a fundamental limit on the estimation accuracy and is a potentially tighter alternative to the Bayesian Cramér-Rao bound.

This paper extends the bounds in [10] and [11] to a weighted sum of MMSEs. Similar to [10] and [11], the bounds derived in this paper are obtained by constraining the input distribution to be $\varepsilon$-close to a Gaussian reference distribution in terms of the KL divergence. The estimators that attain the upper bound are minimax robust against deviations from the assumed input distribution. Finally, the proposed bounds are evaluated for generalized Gaussian distributions and the uniform distribution. It is shown that, in some cases, the lower bounds derived in this paper are tighter than the Cramér–Rao lower bound. Interestingly, the performance of the proposed bounds improves as the dimension of the input vector increases.

The weighted sum of MMSEs arises in various practical applications in signal processing and communications. For example, it has been shown that for a Gaussian prior the MMSE in the time domain can be expressed as a sum of MMSEs in the frequency domain [12]. In multiple input multiple output (MIMO) wireless communications, the MMSE is frequently expressed as a sum of MMSEs or sum of inverse MMSEs of multiple parallel channels [13, 14, 15, 16]. In the context of distributed statistical inference, weighted sums of MMSEs play an important role in parameter estimation problems, where noisy measurements from multiple randomly deployed sensors are used to estimate the parameter of interest. In such scenarios, typically the estimates from the local sensors are fused to obtain a global estimate of the parameter. However, in practice, not much is known about the analytical characterization the optimal performance of the global estimator [17]. The study of the weighted sum of MMSEs reported in this paper is a step in this direction. The weighted sum of MMSEs is not only an appropriate objective function for distributed estimation, since it provides a platform to establish a global performance measure, but it also allows to prioritize sensors by assigning them weights. Thus, a highly informative sensor will be assigned a higher weight in the linear combination of MMSEs of all the sensors. This could have important applications in energy-efficient sensor networking where only highly informative sensors transmit their local decisions, while the sensors deemed less-informative abstain from transmission, thus saving energy and time for decision making [18], [19].

The rest of the paper is organized as follows. In Section II, we provide a mathematical statement of the problem addressed in the paper. The upper and lower bounds on the weighted sum of MMSEs, which are the main results of this work, are stated in Section III. The usefulness of the proposed bounds in distributed estimation is illustrated via an example application and related details are discussed in Section IV. Concluding remarks are provided in Section V.

II Problem Formulation

Let $(\mathbb{R}^{K},\mathcal{B}^{K})$ denote the $K$-dimensional Borel space. Consider $J$ additive-Gaussian-noise channels $Y_{j}=X+N_{j}$, $j=1,\ldots,J$, where $X$ and $N_{1},\ldots,N_{J}$ are independent $(\mathbb{R}^{K},\mathcal{B}^{K})$-valued random variables. All $N_{j}$ are assumed to be zero-mean Gaussian distributed, i.e., $P_{N_{j}}=\mathcal{N}(0,\Sigma_{N_{j}})$, where $\mathcal{N}(\mu,\Sigma)$ denotes the Gaussian distribution with mean $\mu$ and covariance $\Sigma$.

We define the individual MSEs as functions of the estimator $f_{j}$ and the input distribution $P_{X}$, i.e.,

[TABLE]

The individual MMSEs are accordingly defined as

[TABLE]

where $\mathcal{F}$ denotes the set of all all feasible estimators, i.e.,

[TABLE]

The following two optimization problems are investigated in this paper:

[TABLE]

where $\lambda_{1},\ldots,\lambda_{J}>0$ are fixed positive weights and the set of feasible distribution is defined as

[TABLE]

Note that, $\mathcal{P}_{\varepsilon}$ is a KL divergence ball centered at $P_{0}$ of radius $\varepsilon$. As we proceed, it will become clear that it is useful to choose the reference distribution $P_{0}$ to be Gaussian:

[TABLE]

Finally, to allow for a compact notation, the following matrices are introduced:

[TABLE]

where $j=1,\ldots,J$ and $W^{\text{T}}$ denotes the transpose of $W$.

III Main Result

The main result of the paper is provided in the following theorem.

Theorem 1**.**

If $(\alpha^{*},\Sigma_{X}^{*})$, with $\Sigma_{X}^{*}$ positive definite and $\alpha^{*}\geq 0$, solve

[TABLE]

then

[TABLE]

solves problem (1). Analogously, if $(\alpha^{\dagger},\Sigma_{X}^{\dagger})$, with $\Sigma_{X}^{\dagger}$ positive definite and $\alpha^{\dagger}\leq 0$, solve (8) and (9), then

[TABLE]

solves problem (2).

Since the input distributions that attain these bounds are Gaussian of the form P_{X}\sim\mathcal{N}\bigl{(}\mu_{0},\Sigma_{X}\bigr{)}, the individual MMSE estimators in both cases are given by

[TABLE]

where $y_{j}\in\mathbb{R}^{K}$ denotes the observed realization of $Y_{j}$. The corresponding MMSEs are given by

[TABLE]

The lower and upper bounds on the weighted sum of MMSEs in (1) and (2) are then given by

[TABLE]

where $\operatorname{MMSE}_{j}^{*}$ and $\operatorname{MMSE}_{j}^{\dagger}$ are shorthand notations for $\operatorname{MMSE}_{j}$ in (6) evaluated at $\Sigma_{X}^{*}$ and $\Sigma_{X}^{\dagger}$, respectively.

Proof.

The proof of the main result follows along the same lines as the proof in [10, Sec. 4]. Consider the maximization in (1), which can be written as the minimax problem

[TABLE]

where the infimum is taken jointly over all $f_{1},\ldots,f_{J}$. A sufficient condition for $P_{X}^{*}$ and $f_{1}^{*},\ldots,f_{J}^{*}$ to solve (17), and hence (1), is that they satisfy the saddle point conditions [20, Exercise 3.14]

[TABLE]

for all $f_{1},\ldots,f_{J}\in\mathcal{F}$ and

[TABLE]

for all $P_{X}\in\mathcal{P}_{\varepsilon}$. The fact that the estimators $f_{j}$ in (12) minimize the right hand side of (18) follows directly from the definition of the MMSE [21, Chapter 10.4]. In the remainder of the proof, it is shown that $P_{X}^{*}$ in (10) satisfies (19).

First, the right hand side of (19) is written as

[TABLE]

where $h\colon\mathbb{R}^{K}\to\mathbb{R}$ is independent of $P_{X}$ and given by

[TABLE]

with c\coloneqq\sum_{j=1}^{J}\lambda_{j}\operatorname{tr}\bigl{(}(I-W_{j})\Sigma_{N}(I-W_{j})^{\text{T}}\bigr{)} being a constant that is independent of $x$. Using the auxiliary result on bounds on expectations under $f$-divergence constraints in [10, Sec. 4.1], it follows that the density $p_{X}^{*}$ of the optimal distribution $P_{X}^{*}$ is of the form

[TABLE]

where $p_{0}$ denotes the density of the reference distribution $P_{0}$ and $\alpha\geq 0$, $\beta\in\mathbb{R}$ need to be chose appropriately. Substituting $h$ in (25) with (24) and using (4) yields

[TABLE]

where, without loss of generality, $\alpha$ has been scaled by $\tfrac{1}{2}$ and

[TABLE]

Multiplying both sides of (29) by $\Sigma_{X}$ from the left and the right and rearranging the terms yields (8).

Knowing that $P_{X}$ and $P_{0}$ are Gaussian distributions with identical means, their KL divergence is given by

[TABLE]

Equating (30) with $\varepsilon$ yields the optimality condition (9). This concludes the proof of optimality of $P_{X}^{*}$.

The proof of optimality of $P_{X}^{\dagger}$ follows analogously, the only difference being that the sign of $\alpha$ is reversed; relevant details are provided in [10, Sec. 4.1]. ∎

IV An application involving distributed estimation

In this section, we first present an example application involving distributed estimation where the weighted sum of MMSEs is relevant. We show how a simple modification of the conventional distributed processing leads to significant improvements in the performance characterization of practical distributed estimators. We then derive bounds on the weighted sum of MMSEs for distributed estimation with arbitrary distributions in Gaussian noise. Lastly, we specialize these bounds for the generalized Gaussian and uniform distributions. Numerical evaluations, presented to verify the theoretical findings, reveal surprising features of the proposed bounds.

Consider the problem of distributed estimation using wireless sensor networks (WSNs), where $J$ sensors are randomly distributed in the region of interest (ROI). The $j^{\text{th}}$ sensor is located at a distance $d_{j}$ from a target. By “target” we are referring to some activity; for example, fire in the ROI. The target’s signal power is assumed to follow the isotropic power attenuation model [22]. The signal power at sensor $j$ is given by $\rho_{j}^{2}=\rho_{0}^{2}/(1+\gamma d_{j}^{m})$, $j=1,\dots,J$, where $\rho_{0}^{2}$ is the target’s signal power at distance zero, $m$ is the signal decay exponent taking values between 2 and 3, and $\gamma$ is a constant (larger $\gamma$ implies faster power decay). In practice, the parameters $m$ and $\gamma$ pertaining to the wireless medium are obtained by performing experiments before the WSN is deployed, though uncertainty is associated with this knowledge. For the purpose of this paper, let us consider the simple goal of estimating the distances $d_{j}$ based on the noisy observations made by the sensors. The knowledge of $d_{j}$ is typically used to infer the presence/absence of a target in the ROI (see [22]). In conventional distributed estimation, the estimates of $d_{j}$ computed by the $j^{\text{th}}$ local sensor is transmitted to a central processing unit, which aggregates $d_{j}$, $j=1,\dots,J$ to compute a system-level estimate of the distance vector $\hat{\mathbf{d}}=(\hat{d}_{1},\dots,\hat{d}_{J})$. However, to the best of our knowledge, theoretical insights into the system-level estimator’s accuracy are lacking in the literature.

Let us now consider a simple modification to the above scheme. Instead of the local estimates, if the $J$ sensors transmit their local MSEs to the central unit, then a weighted sum of MMSEs can be thoroughly analyzed using the findings of this paper. This new scheme provides a comprehensive view of the performance of system-level (or, global) estimators unlike existing distributed estimation wherein the local estimates are simply fused at the control unit without insightful performance guarantees. Our study provides engineering guidelines for the design and analysis of large-scale sensor networks which are important components in several critical infrastructures like the Smart Grid, IoT and other cyber-physical systems. Possible improvements in global system performance are demonstrated with the following two examples. We first derive upper and lower bounds on the linear combination of MMSEs for arbitrary distributions in Gaussian noise and then derive these bounds for the generalized Gaussian distribution.

IV-A Bounds on the linear combinations of MMSEs for arbitrary distributions in Gaussian noise

Consider the linear combination of MMSEs for an arbitrary distribution $P_{X}$ such that

[TABLE]

For a given $P_{X}$, upper and lower bounds on the linear combination of MMSEs can be derived by the following steps:

Find the best Gaussian approximation of $P_{X}$ in terms of the KL divergence. In other worlds, find a Gaussian $Q$ that minimizes $D_{\text{KL}}(P_{X}\|Q)$ and compute $\varepsilon=D_{\text{KL}}(P_{X}\|Q)$. 2. 2.

Use the value of $\varepsilon$ found in Step 1 to compute the upper and lower bounds in (1) and (2), respectively.

We can evaluate the effectiveness of this procedure by comparing it to the bounds attained by individually bounding each term in the linear combination. The most popular bounds on the individual MMSEs are the following:

[TABLE]

where $J(P_{X})$ is the Fisher information of $P_{X}$. The upper bound in (32) is obtained by using the best linear estimator instead of the optimal estimator. The lower bound in (33) is the Cramér–Rao lower bound. We refer to the bounds obtained by bounding individual MMSEs as local and the bounds that work directly on the linear combination as global.

IV-B Generalized Gaussian and Uniform distributions

Let us now consider distributions that are either concentrated or heavy-tailed. A classic example is the generalized Gaussian distribution, whose density is given by

[TABLE]

where $c_{p}$ is the normalization constant. The covariance matrix, Fisher information and the best Gaussian approximation for this distribution are given by

[TABLE]

To evaluate the performance of our bounds we set $K=3$, $J=4$, and

[TABLE]

and compare the resulting bounds in Fig. 1. Specifically, Fig. 1 comprises the following bounds:

lower bounds obtained via (2) (solid black line); 2. 2.

upper bounds obtained via (1) (dotted black line); 3. 3.

local upper bounds attained via (32) (solid gray line); 4. 4.

local lower bounds attained via (33) (dashed gray line); 5. 5.

local lower bounds attained via (2) (dashed-dotted black line). The local version of the lower bound in (2) is obtained by individually minimizing each MMSE with the same KL constraint. Hence, the resulting solution is independent of $\lambda_{i}$’s; and 6. 6.

local upper bounds attained via (1) (dashed black line). The local version of the upper bound in (1) is obtained by individually maximizing each MMSE with the same KL constraint.

Another important feature of the proposed bounds is that they hold for prior distributions that do not necessarily have a well-defined Fisher information. Note that, as a consequence of Stem’s inequality, the finiteness of Fisher information implies finite $D_{\text{KL}}(P_{X}\|Q)$. The converse statement, however, is not true, and there are distributions without well-defined Fisher information but with finete $D_{\text{KL}}(P_{X}\|Q)$. The practical implications of this phenomenon is that, our bounds hold for a larger set of distributions. This property of the proposed bounds has been discussed in detail in [11]. An example of such a prior distribution is a uniform distribution over a $K$-ball. We demonstrate this in Fig. 2, where the plots of our lower and upper bounds versus the radius of the $K$-ball using parameters in (39) is shown. It is interesting to observe that the larger the radius of the $K$-ball the better is the performance of the proposed lower and upper bounds.

V Concluding remarks

Upper and lower bounds on the weighted sum of MMSEs for additive Gaussian noise channels have been derived. It has been shown that these bounds take the coupling between the individual MMSEs into account and thereby are significantly tighter than the existing bounds. Examples have been provided to show how the presented results can be particularly useful for the design and analysis of sensor networks for distributed estimation, where the weighted sum of MMSEs can be used to monitor the performance, establish operating regions and to design robust local estimators. Further insights into the robustness of the local estimators for distributed setups require a more careful analysis and will be subject of future research.

Acknowledgment

The work of A. Dytso and H. V. Poor was supported by the U. S. National Science Foundation under Grant CCF–1513915. The work of Nagananda K. G. was supported by the European Research Council under grant agreement 715111.