Estimating Noisy Order Statistics

TL;DR

This paper introduces an estimation framework for assessing sorting accuracy on noisy data, revealing properties of MMSE growth and proposing practical estimators with performance guarantees.

Contribution

It develops a novel MMSE-based framework for noisy order statistics, deriving optimal and suboptimal estimators and analyzing their performance under symmetry conditions.

Findings

MMSE grows sublinearly with data size

Optimal estimator is a linear combination of unsorted data estimators

Common bounds like Bayesian Cramér-Rao are not suitable for this problem

Abstract

This paper proposes an estimation framework to assess the performance of sorting over perturbed/noisy data. In particular, the recovering accuracy is measured in terms of Minimum Mean Square Error (MMSE) between the values of the sorting function computed on data without perturbation and the estimator that operates on the sorted noisy data. It is first shown that, under certain symmetry conditions, satisfied for example by the practically relevant Gaussian noise perturbation, the optimal estimator can be expressed as a linear combination of estimators on the unsorted data. Then, two suboptimal estimators are proposed and performance guarantees on them are derived with respect to the optimal estimator. Finally, some surprising properties on the MMSE of interest are discovered. For instance, it is shown that the MMSE grows sublinearly with the data size, and that commonly used MMSE lower bounds such as the Bayesian Cram\'er-Rao and the maximum entropy bounds either cannot be applied or are not suitable.

Figures

Equations

$$\mathbf{P}_{\{1,2,3\}}=\mathbf{I}_{3},\qquad\text{and}\qquad\mathbf{P}_{\{2,3,1\}}=\left[\begin{array}[]{ccc}0&1&0\\ 0&0&1\\ 1&0&0\end{array}\right].$$

$$(U_1,U_2,\dots ,U_n)\stackrel{d}{=}(U_{{\pi }_1},U_{{\pi }_2},\dots ,U_{{\pi }_n}),$$

$$f_{\mathbf{Y}|\mathbf{X}}(\mathbf{y}|\mathbf{x})=\prod _{i=1}^n{f}_{Y_i|X_i}(y_i|x_i).$$

$$\mathrm{mmse}(\vec{\mathbf{X}}|\vec{\mathbf{Y}})=\mathbb{E}\left[{\Vert \vec{\mathbf{X}}-\mathbb{E}\left[\vec{\mathbf{X}}|\vec{\mathbf{Y}}\right]\Vert }^2\right].$$

$${\mathbb{E}}_{{\Pi }_1,{\Pi }_2}\left[f_{\mathbf{Y}|\mathbf{X}}({\mathbf{P}}_{{\Pi }_2}\mathbf{y}|{\mathbf{P}}_{{\Pi }_1}\mathbf{x})\right]={\mathbb{E}}_{\Pi }\left[f_{\mathbf{Y}|\mathbf{X}}({\mathbf{P}}_{\Pi }\mathbf{y}|\mathbf{x})\right],$$

$$\frac{\mathbb{E}[\vec{\mathbf{X}}|\vec{\mathbf{Y}}=\vec{\mathbf{y}}]}{n!}=\sum _{\pi \in \mathcal{P}}\frac{f_{\mathbf{Y}}({\mathbf{P}}_{\pi }\vec{\mathbf{y}})}{f_{\vec{\mathbf{Y}}}(\vec{\mathbf{y}})}\mathbb{E}\left[\mathbf{X}\cdot 1_{{\vec{\mathbb{R}}}_n}(\mathbf{X})|\mathbf{Y}={\mathbf{P}}_{\pi }\vec{\mathbf{y}}\right].$$

$$\mathbb{E}[\vec{\mathbf{X}}|\vec{\mathbf{Y}}=\vec{\mathbf{y}}]$$

$$\mathbb{E}\left[X_{(k)}|\vec{\mathbf{Y}}=\vec{\mathbf{y}}\right]$$

$$=$$

$$\stackrel{(\mathrm{a})}{=}$$

$$=$$

$$\cdot f_{\mathbf{Y}|\mathbf{X}}({\mathbf{P}}_{\pi }\vec{\mathbf{y}}|\mathbf{t})\mathrm{d}\mathbf{t}$$

$$\stackrel{(\mathrm{b})}{=}$$

$$=$$

$$f_{\vec{\mathbf{X}}|\vec{\mathbf{Y}}}(\vec{\mathbf{x}}|\vec{\mathbf{y}})=\frac{n!}{f_{\vec{\mathbf{Y}}}(\vec{\mathbf{y}})}{f}_{\mathbf{X}}(\vec{\mathbf{x}})\sum _{\pi \in \mathcal{P}}{f}_{\mathbf{Y}|\mathbf{X}}({\mathbf{P}}_{\pi }\vec{\mathbf{y}}|\vec{\mathbf{x}})$$

$$\frac{f_{\mathbf{Y}}({\mathbf{P}}_{\pi }\vec{\mathbf{y}})}{f_{\vec{\mathbf{Y}}}(\vec{\mathbf{y}})}=\frac{f_{\mathbf{Y}}(\vec{\mathbf{y}})}{f_{\vec{\mathbf{Y}}}(\vec{\mathbf{y}})}=\frac{1}{n!},$$

$$\mathbb{E}\left[X_k\cdot 1_{{\vec{\mathbb{R}}}_n}(\mathbf{X})|\mathbf{Y}={\mathbf{P}}_{\pi }\vec{\mathbf{y}}\right]$$

$$=$$

$${\hat{f}}_k(\vec{\mathbf{y}})=\sum _{\pi \in \mathcal{P}}\mathbb{E}\left[X_k|\mathbf{Y}={\mathbf{P}}_{\pi }\vec{\mathbf{y}}\right]\cdot \mathbb{P}\left[\mathbf{X}\in {\vec{\mathbb{R}}}_n|\mathbf{Y}={\mathbf{P}}_{\pi }\vec{\mathbf{y}}\right],$$

$$\hat{\mathbf{f}}(\vec{\mathbf{y}})=[{\hat{f}}_1(\vec{\mathbf{y}}),\dots ,{\hat{f}}_n(\vec{\mathbf{y}})].$$

$$\Delta =\left|\mathbb{E}\left[{\Vert \vec{\mathbf{X}}-\hat{\mathbf{f}}(\vec{\mathbf{Y}})\Vert }^2\right]-\mathrm{mmse}(\vec{\mathbf{X}}|\vec{\mathbf{Y}})\right|.$$

$$\Delta \le {\Delta }_{\text{up}}=\sum _{\pi \in \mathcal{P}}\mathbb{E}\left[\Vert \mathbf{X}{\Vert }^{2}g\left({\mathbf{P}}_{\pi }\mathbf{Y}\right)\mid \mathbf{Y}\in {\vec{\mathbb{R}}}_n\right],$$

$$g(\mathbf{y})=\mathbb{P}\left[\mathbf{X}\in {\vec{\mathbb{R}}}_n|\mathbf{Y}=\mathbf{y}\right]\left(1-\mathbb{P}\left[\mathbf{X}\in {\vec{\mathbb{R}}}_n|\mathbf{Y}=\mathbf{y}\right]\right).$$

$${\hat{h}}_k(\vec{\mathbf{y}})=\sum _{\pi \in \mathcal{P}}\mathbb{E}[X_k|\mathbf{X}\in {\vec{\mathbb{R}}}_n]\cdot \mathbb{P}[\mathbf{X}\in {\vec{\mathbb{R}}}_n|\mathbf{Y}={\mathbf{P}}_{\pi }\vec{\mathbf{y}}],$$

$$\hat{\mathbf{h}}(\vec{\mathbf{y}})=[{\hat{h}}_1(\vec{\mathbf{y}}),\dots ,{\hat{h}}_n(\vec{\mathbf{y}})].$$

$${\hat{h}}_k(\vec{\mathbf{y}})$$

$$\mathbb{E}\left[{\Vert \vec{\mathbf{X}}-\hat{\mathbf{h}}(\vec{\mathbf{Y}})\Vert }^2\right]=\mathbb{E}\left[{\Vert \vec{\mathbf{X}}-\mathbb{E}[\vec{\mathbf{X}}]\Vert }^2\right]=\mathsf{Var}(\vec{\mathbf{X}}).$$

$$\underset{t\to \infty }{\lim }\mathbb{E}[\mathbf{X}|{\mathbf{Y}}_t]=\mathbb{E}[\mathbf{X}]\text{ a.s.}$$

$${\Delta }_t=\left|\mathbb{E}\left[{\Vert \vec{\mathbf{X}}-\hat{\mathbf{h}}({\vec{\mathbf{Y}}}_t)\Vert }^2\right]-\mathrm{mmse}(\vec{\mathbf{X}}|{\vec{\mathbf{Y}}}_t)\right|.$$

$$\underset{t\to \infty }{\lim }{\Delta }_t=0.$$

$${\hat{\mathbf{\ell }}}_{\text{MLE}}(\vec{\mathbf{y}})\in \arg \underset{\mathbf{t}\in {\vec{\mathbb{R}}}_n}{\max }\sum _{\pi \in \mathcal{P}}{f}_{\mathbf{Y}|\mathbf{X}}({\mathbf{P}}_{\pi }\vec{\mathbf{y}}|\mathbf{t}).$$

$${\hat{f}}_k(\vec{\mathbf{y}})=\sum _{\pi \in \mathcal{P}}{a}_{\pi }\left(\vec{\mathbf{y}}\right){\left[{\mathbf{P}}_{\pi }\vec{\mathbf{y}}\right]}_k,$$

$$a_{\pi }\left(\vec{\mathbf{y}}\right)=\frac{\sigma }{1+{\sigma }^2}\mathbb{P}\left[\mathbf{X}\in {\vec{\mathbb{R}}}_n|\mathbf{Y}={\mathbf{P}}_{\pi }\vec{\mathbf{y}}\right].$$

$$\underset{\sigma \to 0}{\lim }{\Delta }_{\text{up}}$$

Full text

I Introduction

Today, the sorting function is widely used in a broad variety of applications. For instance, it is well recognized that sorting is a benchmark for several modern recommender and distributed computing systems (e.g., Hadoop MapReduce).

In this paper, we focus on analyzing the performance of the sorting function over data that is perturbed. Such a data perturbation might occur because of data privacy purposes, i.e., the data is sensitive (e.g., clinical/genomic health) and hence it is required to remain confidential/private. This gives rise to a natural question: How does data perturbation affect the performance of the sorting function?

Our main objective is to propose and make first steps in analyzing an estimation framework, which seeks to answer the question above. Towards this end, we discover several properties that shed light on the effect of the sorting function on the data statistics.

I-A Related Work

In this work, we leverage tools and properties from the theory of order statistics. Order statistics indeed represents an indispensable tool in the modern theory of statistical inference. Next, we briefly mention some notable examples.

The estimation of parameters of a family of distributions from an ordered vector has received a considerable attention. For example, for a location-scale family of distributions with density function $\frac{1}{\sigma}f\left(\frac{x-\mu}{\sigma}\right)$, the best linear unbiased estimator (BLUE) of $\sigma$ and $\mu$ has been found in [1]. In particular, the BLUE was shown to be a function of only the covariance matrix and the mean vector of the order statistics. However, note that the computation of the covariance matrix and the mean vector of the order statistics is often a formidable task. Explicit expressions for moments of order statistics are in fact known only for some specific distributions and have been tabulated in [2]. A rich body of literature also exists on universal bounds on moments of order statistics [3].

Having observed a partial sample of the first $r$ order statistics from a sample of size $n$, the best linear unbiased predictor (BLUP) of the remaining $n-r$ terms has been characterized in [4]. Specifically, the predictor in [4] was shown to depend only on the covariance matrix and the mean of the order statistics. Interesting connections between the BLUE and BLUP have been derived in [5].

Due to its application to life-testing experiments, inference of censored (i.e., incomplete data) order statistics has also received attention. Interestingly, for various types of censoring protocols, closed-form expressions for maximum likelihood estimators (MLEs) of parameters are available. A comprehensive survey of several censoring scenarios can be found in [6].

Order statistics also appears in the study of outliers since these are expected to be a few extreme order statistics. Several effective tests are formed from extreme order statistics that seek to compute the deviation of the candidate outliers from the rest of the data [7]. Another application of order statistics is on the goodness-of-fit tests. The most classical example of such a test is the Shapiro and Wilk’s test for normality [8].

A collection of articles summarizing applications, results and historic perspectives on order statistics can be found in [9].

While classical works have focused on estimating parameters of distributions from order statistics or recovering incomplete ordered sequences, modern applications have focused on sorting with noisy data. In [10], the authors considered an additive noise linear regression model in which the output variables are permuted and might be incorrectly associated with the input variables. The goal is to estimate the correct permutation matrix and the linear parameter. Further extensions of [10] include [11] where the authors considered an additive noise regression model with the goal of estimating an unknown monotone increasing function of the input variables.

There is also a large body of works in computer science that focuses on sorting with unreliable machines that perform pairwise comparisons; see [12, 13] and references therein.

Different from the aforementioned works, we are here interested in estimating the values of the components of the original sorted vector, by performing joint estimation and sorting. Moreover, we focus on the Bayesian setting, i.e., we assume a prior distribution on the input variables. This work builds on our preliminary results in [14], and extends them by studying the effect of sorting on the data statistics.

I-B Contributions

In this work, we focus on assessing the performance of the sorting function over data that is perturbed. Our goal is to quantify the performance loss of the sorting function versus different levels of noise perturbation. Our main contributions can be summarized as follows:

We pose the problem within an estimation framework. In particular, as a metric, we adopt the Minimum Mean Square Error (MMSE) between the values of the sorting function computed on the original data (i.e., with no perturbation) and the estimation that uses the sorting function applied to the noisy version of the data. Our proposed framework is presented in Section II. 2. 2.

We analyze the optimal estimator, i.e., the conditional expectation of observing the original data as sorted, given the observation of the noisy sorted data. We show that, under certain symmetry conditions, satisfied for example by the practically relevant Gaussian noise perturbation, the optimal estimator can be expressed as a linear combination of estimators of the unsorted data from unsorted noisy observations. Our analysis on the optimal estimator can be found in Section III. 3. 3.

We leverage the structure of the optimal estimator to propose two suboptimal estimators, and we analyze their performance with respect to the optimal estimator. We prove that, with Gaussian statistics, one estimator is asymptotically optimal in the low noise regime, whereas the other is asymptotically optimal in the high noise regime. We also characterize the structure of the MLE, and we numerically assess its performance with Gaussian statistics. Our analysis on the suboptimal estimators can be found in Section IV and Section V. 4. 4.

We discover several properties on the MMSE of interest, which highlight interesting observations on how the sorting function affects the statistics. For instance, in Section VI we show that for order statistics the MMSE grows sublinearly with the size of the vectors; this represents a fundamental difference from unsorted data for which the MMSE grows linearly in the size of the vector. Another surprising observation concerns lower bounds on the MMSE of interest. In Section VII, we indeed show that the Bayesian Cramér-Rao bound cannot be applied, and the maximum entropy bound does not scale properly with the size of the vectors.

II Notation, Definitions and System Model

Boldface upper case letters $\mathbf{X}$ denote vector random variables; the boldface lower case letter $\bf{x}$ indicates a specific realization of $\mathbf{X}$; $\vec{\mathbf{X}}$ is $\mathbf{X}$ ordered in ascending order; $X_{i}$ specifies the $i$-th entry of ${\mathbf{X}}$ and $X_{(i)}$ indicates the $i$-th entry of $\vec{\mathbf{X}}$; $\|\mathbf{x}\|$ denotes the $\ell_{2}$-norm of the vector $\mathbf{x}$; $\mathbf{I}_{n}$ is the identity matrix of dimension $n$; $\mathbf{0}_{n}$ is the column vector of dimension $n$ of all zeros; $\delta(x)$ is the Dirac delta function; $1_{\{S\}}(x_{i})$ is the indicator function over the set $S$; $[n]$ indicates the set of integers $\{1,\dots,n\}$. The set $\vec{\mathbb{R}}_{n}=\{{\bf x}:\vec{\bf x}\in\mathbb{R}_{n}\}$; $\mathcal{P}$ denotes the collection of all permutations of the elements of $[n]$ and $|\mathcal{P}|=n!$. $\mathbf{P}_{\pi}$ where $\pi\in\mathcal{P}$ is a permutation matrix associated with $\pi$. For example, for $n=3$

[TABLE]

We next provide a definition, which will be used in the proof of our main results.

Definition 1**.**

A sequence of random variables $U_{1},U_{2},\ldots,U_{n}$ is said to be exchangeable if for any permutation $(\pi_{1},\pi_{2},\ldots,\pi_{n})$ of the indices $[n]$, we have that

[TABLE]

where $\stackrel{{\scriptstyle d}}{{=}}$ denotes equality in distribution.

Note that any convex combination or mixture distribution of independent and identically distributed sequences of random variables is exchangeable.

We consider the framework shown in Fig. 1, where an $n$-dimensional random vector $\mathbf{X}$ is generated according to a certain probability density function (PDF) $f_{\mathbf{X}}(\cdot)$ and then passed through a noisy channel with a transition PDF equal to $f_{\mathbf{Y}|\mathbf{X}}(\cdot|\cdot)$. This is assumed to be a parallel channel, i.e.,

[TABLE]

The output of the channel – denoted as $\mathbf{Y}$ – is finally sorted in ascending order, i.e.,$\vec{\mathbf{Y}}$ is the sorted version of $\mathbf{Y}$.

In this work, we are interested in characterizing the MMSE of estimating $\vec{\mathbf{X}}$ – which denotes the sorted version of ${\mathbf{X}}$ (i.e., the ground truth) – when $\vec{\mathbf{Y}}$ is observed.

III On the Optimal MMSE Estimator

The objective of this section is to study the MMSE of estimating $\vec{\mathbf{X}}$ from an observation $\vec{\mathbf{Y}}$. The MMSE is given by

[TABLE]

It is well known that the conditional expectation $\mathbb{E}\left[\vec{\mathbf{X}}|\vec{\mathbf{Y}}\right]$ is the optimal estimator under the square error criterion. The next theorem provides a characterization of $\mathbb{E}\left[\vec{\mathbf{X}}|\vec{\mathbf{Y}}\right]$ in terms of the distribution of $({\bf X},{\bf Y})$ under certain symmetry conditions.

Theorem 1**.**

Let $({\bf X},{\bf Y})$ be continuous random vectors. Assume that ${\bf X}$ is exchangeable and that, $\forall(\mathbf{x},\mathbf{y})$

[TABLE]

where $(\Pi,\Pi_{1},\Pi_{2})$ are mutually independent and uniformly distributed on $\mathcal{P}$. Then,

[TABLE]

In addition, if ${\bf Y}$ is exchangeable, then

[TABLE]

Proof:

We have

[TABLE]

where the labeled equalities follow from: $\rm{(a)}$ the identity

[TABLE]

in Lemma 16 in Appendix A; and $\rm{(b)}$ using Bayes’ rule. This concludes the proof of (3). To show that (4) holds observe that, if ${\bf Y}$ is exchangeable, then

[TABLE]

where the last equality follows by using Lemma 15 in Appendix A. This concludes the proof of Theorem 1. ∎

Remark 1**.**

Examples of noisy transformations that satisfy (2) include the practically relevant Gaussian distribution.

Theorem 1 allows to express the optimal estimator of elements of $\vec{\mathbf{X}}$ from an observation $\vec{\mathbf{Y}}$ as a linear combination of estimators of the original unsorted ${\bf X}$ from the unsorted channel observation ${\bf Y}$. This has the potential of significantly simplifying the computation of the conditional expectation as there is no need to find the joint distribution of $(\vec{\mathbf{X}},\vec{\mathbf{Y}})$.

In the remaining part of the paper, we assume that ${\bf X}$ and ${\bf Y}$ are both exchangeable, and hence we focus on further analyzing the optimal estimator in (4).

IV Suboptimal Estimators

Observe that the conditional expectation in (4) can be equivalently written as follows:

[TABLE]

We now use the identity above to propose and study two different suboptimal estimators. The idea is to approximate (6) by dropping certain dependencies on $\mathbf{Y}$ or $\mathbf{X}\in\vec{\mathbb{R}}_{n}$. Moreover, we also discuss the structure of the MLE.

The first suboptimal estimator of $X_{(k)}$ that we analyze is

[TABLE]

Observe that the only difference between the optimal estimator in (6) and the suboptimal estimator in (7a) is that in the latter the conditioning on $\mathbf{X}\in\vec{\mathbb{R}}_{n}$ has been dropped. In other words, we are implicitly using the approximation $\mathbb{E}\left[X_{k}|\mathbf{X}\in\vec{\mathbb{R}}_{n},\mathbf{Y}\right]\approx\mathbb{E}\left[X_{k}|\mathbf{Y}\right]$.

The next theorem, whose proof can be found in Appendix B, compares the performance of the optimal estimator in (4) to the proposed suboptimal estimator in (7a).

Theorem 2**.**

Assume that ${\bf X}$ and ${\bf Y}$ are exchangeable and that the assumption in (2) holds. Let

[TABLE]

Then,

[TABLE]

*where *

[TABLE]

Intuitively, the estimator in (7a) performs well when ${\bf X}$ and ${\bf Y}$ have a high degree of correlation, and the information contained into ${\bf Y}$ is enough to produce a good estimate of $\vec{{\bf X}}$. This scenario typically occurs when the noise between ${\bf X}$ and ${\bf Y}$ is weak. However, this estimator starts performing poorly when ${\bf X}$ and ${\bf Y}$ are only weakly dependent on each other. In this case, in fact, ignoring the condition $\mathbf{X}\in\vec{\mathbb{R}}_{n}$ can lead to a severe penalty. Motivated by this observation, we next propose an estimator that MMSE wise works well in the opposite regime, that is when ${\bf X}$ and ${\bf Y}$ are weakly dependent.

The second suboptimal estimator of $X_{(k)}$ that we analyze is

[TABLE]

Observe that the only difference between the optimal estimator in (6) and the suboptimal estimator in (11a) is that in the latter the conditioning on $\mathbf{Y}$ inside the conditional expectation has been dropped. In other words, we are implicitly using $\mathbb{E}\left[X_{k}|\mathbf{X}\in\vec{\mathbb{R}}_{n},\mathbf{Y}\right]\approx\mathbb{E}\left[X_{k}|\mathbf{X}\in\vec{\mathbb{R}}_{n}\right]$. Before analyzing the performance of the suboptimal estimator given in (11), we state the next lemma (whose proof is provided in Appendix C), which offers a simplified expression for the estimator in (11).

Lemma 3**.**

Assume that ${\bf X}$ and ${\bf Y}$ are exchangeable random vectors. Then, the estimator in (11) can be written as

[TABLE]

In other words, Lemma 3 shows that by dropping the conditioning on ${\mathbf{Y}}$ inside the conditional expectation in the optimal estimator in (6), then the resulting suboptimal estimator only uses the knowledge of the prior distribution of $\vec{\mathbf{X}}$. The optimal MSE of such an estimator can be easily computed as

[TABLE]

Intuitively, such an estimator performs well when the output ${\bf Y}$ contains little information about the input ${\bf X}$, e.g., when the noise between ${\bf X}$ and ${\bf Y}$ is high. Sufficient conditions for the MSE optimality of this estimator are given in the next theorem, whose proof can be found in Appendix D.

Theorem 4**.**

Assume that for every $t\in\mathbb{N}$, ${\bf X}$ and ${\bf Y}_{t}$ are exchangeable random vectors, and that the assumption in (2) holds. In addition, assume that

[TABLE]

and let

[TABLE]

Then, if $\mathbb{E}[\|{\bf X}\|^{2}]<\infty$, we have that

[TABLE]

Remark 2**.**

The assumption in (14) holds for several channels such as the practically relevant Gaussian noise channel. This case will be studied in more detail in Section V. Another important feature of Theorem 4 is that the conditions for the convergence depend only on $({\bf X},{\bf Y})$ and not on $(\vec{\mathbf{X}},\vec{\mathbf{Y}}).$

We now conclude this section by characterizing the structure of the MLE of $\vec{\mathbf{X}}$ from $\vec{\mathbf{Y}}$ in terms of the conditional distribution of the original unsorted pair $({\bf X},{\bf Y})$. By leveraging Lemma 16 in Appendix A, we obtain

[TABLE]

V Suboptimal Estimators Evaluated with Gaussian Statistics

In this section, we consider the practically relevant case of Gaussian noise, i.e., we assume that ${\bf Y}|{\bf X}=\mathbf{x}~{}\sim\mathcal{N}({\bf x},\sigma^{2}\mathbf{I}_{n})$. In particular, we assess the performance of the two suboptimal estimators in (7) and in (11), as well as of the MLE in (17), by comparing them with the optimal estimator in Theorem 1.

We start by noting that, under the assumption that ${\bf X}\sim\mathcal{N}({\bf 0}_{n},\mathbf{I}_{n})$, the proposed suboptimal estimator in (7a) becomes

[TABLE]

where $\left[\mathbf{P}_{\pi}\vec{\mathbf{y}}\right]_{k}$ is $k$-th component of the vector $\mathbf{P}_{\pi}\vec{\mathbf{y}}$ and where

[TABLE]

Fig. 2 compares the performance, in terms of the MSE, of the suboptimal estimator in (18) (dashed-dotted curve) and the optimal estimator in (4) (solid curve), versus different values of $\sigma$. From Fig. 2 we observe that the suboptimal estimator in (18) performs closely to the optimal estimator in (4) for small values of $\sigma$. However, for higher values of $\sigma$, Fig. 2 suggests that the MSE of the suboptimal estimator in (18) is to within a constant gap of the MSE of the optimal estimator. We next formalize these observations in Theorem 5.

Theorem 5**.**

Let ${\bf X}$ be exchangeable and ${\bf Y}|{\bf X}=\mathbf{x}\sim\mathcal{N}({\bf x},\sigma^{2}\mathbf{I}_{n})$. Assume that $\mathbb{E}[X_{k}^{2}]<\infty$ for all $k\in[n]$. Then, the approximation error in (9) satisfies the following

[TABLE]

Proof:

We start by noting that, by a simple application of the dominated convergence theorem, we have that

[TABLE]

Now observe that

[TABLE]

where the labeled equalities follow from: ${\rm{(a)}}$ using the dominated convergence theorem with the bound

[TABLE]

with $p_{\vec{\mathbf{X}}|\mathbf{Y}}(\mathbf{P}_{\pi}\vec{\mathbf{y}})=\mathbb{P}\left[\mathbf{X}\in\vec{\mathbb{R}}_{n}|\mathbf{Y}=\mathbf{P}_{\pi}\vec{\mathbf{y}}\right],$ and since $\mathbb{E}[X_{k}^{2}]<\infty$; ${\rm{(b)}}$ using the limit in (21); and ${\rm{(c)}}$ using the fact that $1_{\vec{\mathbb{R}}_{n}}\left(\mathbf{P}_{\pi}\mathbf{X}\right)\left(1-1_{\vec{\mathbb{R}}_{n}}\left(\mathbf{P}_{\pi}\mathbf{X}\right)\right)=0$. Combining (23) with the definition of $\Delta_{\text{up}}$ in (9) we obtain (20a).

We now focus on the case of $\sigma\to\infty$, and we obtain

[TABLE]

where the labeled equalities follow from: ${\rm{(a)}}$ the fact that when $\sigma\to\infty$ ${\bf X}$ and ${\bf Y}$ are independent; ${\rm{(b)}}$ using the dominated convergence theorem with the bound in (24) and the assumption that $\mathbb{E}[X_{k}^{2}]<\infty$; and ${\rm{(c)}}$ using the limit in (22). Combining (25) with $\Delta_{\text{up}}$ in (9), we obtain (20b) .

This concludes the proof of Theorem 5. ∎

As highlighted above, Theorem 5 shows that the suboptimal estimator in (18) is asymptotically optimal in the low noise regime, and its MSE performance is always to within a constant gap of the optimal MMSE. The plot of the upper bound on the penalty in (9) for the Gaussian input ${\bf X}\sim\mathcal{N}({\bf 0}_{n},{\bf I}_{n})$ is shown in Fig. 3, versus different values of $\sigma$.

We now turn our attention to analyzing the performance of the suboptimal estimator in (12). In particular, we start by showing that this estimator is optimal in the high noise regime, as stated in the following theorem.

Theorem 6**.**

Let ${\bf X}$ be exchangeable and ${\bf Y}|{\bf X}=\mathbf{x}\sim\mathcal{N}({\bf x},\sigma^{2}\mathbf{I}_{n})$. Assume that $\mathbb{E}[X_{k}^{2}]<\infty$ for all $k\in[n]$. Then, the approximation error in (15) satisfies the following:

[TABLE]

Proof:

To show (26), we have to verify that the limit in (14) holds. For fixed values of $k$ and ${\bf y}$, we have

[TABLE]

where the labeled equalities follow from: ${\rm(a)}$ the fact that ${\bf Y}$ only depends on $X_{k}$ through $Y_{k}$; ${\rm(b)}$ using Bayes rule; and ${\rm(c)}$ using $\mathrm{e}^{-\frac{(y-X_{k})^{2}}{2\sigma^{2}}}\leq 1$ to apply the dominated convergence theorem. This concludes the proof of Theorem 6. ∎

In Fig. 2, we compare the performance of the suboptimal estimator in (12) (dotted curve) and the optimal one in (4) (solid curve), versus different values of $\sigma$. From Fig. 2 we observe that, as also proved in Theorem 6, the suboptimal estimator in (12) performs closely to the optimal estimator in (4) for high values of $\sigma$. We also highlight that, as shown in (13), the MSE of the suboptimal estimator in (12) is simply the variance of $\vec{{\bf X}}$, i.e., it is a constant with respect to $\sigma$.

We conclude this section by analyzing the MLE in (17), whose structure for the Gaussian noise channel is given next.

Lemma 7**.**

Let ${\bf Y}|{\bf X}=\mathbf{x}\sim\mathcal{N}({\bf x},\sigma^{2}\mathbf{I}_{n})$. Then, the MLE is given by

[TABLE]

If the set above is empty, then set ${\bf\hat{\bm{\ell}}}_{\text{MLE}}(\vec{\mathbf{y}})=\vec{\mathbf{y}}$.

Proof:

Specializing (17) to the Gaussian noise channel, a maximum in (17) must satisfy the following equation

[TABLE]

for $\mathbf{t}\in\vec{\mathbb{R}}_{n}$. The expression above is equivalent to

[TABLE]

The proof is concluded by observing that $\|\mathbf{P}_{\pi}\vec{\mathbf{y}}\|=\|\mathbf{y}\|$,

[TABLE]

and canceling the same terms. ∎

In Fig. 2, the performance of the MLE (dashed curve) in Lemma 7 is compared to the other two suboptimal estimators, and the optimal estimator. From Fig. 2 it appears that the MLE is a suitable estimator only when $\sigma\to 0$. The reason is that when $\sigma\to 0$, it can be shown that ${\bf\hat{\bm{\ell}}}_{\text{MLE}}(\vec{\mathbf{y}})=\vec{\mathbf{y}}$, which is an optimal estimator of $\vec{\mathbf{X}}$ since $\lim_{\sigma\to 0}\vec{\mathbf{Y}}=\vec{\mathbf{X}}$ almost surely.

VI Large-Dimension Regime

In this section, we analyze the behavior of the MMSE of estimating $\vec{\mathbf{X}}$ from an observation $\vec{\mathbf{Y}}$ as $n\to\infty$. In particular, our main result is provided in the next theorem, whose proof is presented throughout this section.

Theorem 8**.**

Let ${\bf X}\sim\mathcal{N}(\mathbf{0}_{n},\mathbf{I}_{n})$. Then, we have

[TABLE]

The behavior of $\mathsf{Var}(\vec{\mathbf{X}})/n$ versus different values of $n$ is shown in Fig. 2(d). From Theorem 8, we can readily obtain the behavior of the MMSE as $n\to\infty$, which is stated next.

Corollary 9**.**

Let ${\bf X}\sim\mathcal{N}(\mathbf{0}_{n},\mathbf{I}_{n})$, and let ${\bf Y}|{\bf X}=\mathbf{x}\sim\mathcal{N}({\bf x},\sigma^{2}\mathbf{I}_{n})$. Then, we have

[TABLE]

Proof:

First note that

[TABLE]

Moreover, from the definition of MMSE, we have that

[TABLE]

where the inequality follows by using $\mathbb{E}\left[\vec{\mathbf{X}}\right]$ as suboptimal estimator. This implies that

[TABLE]

By replacing $\vec{\mathbf{Y}}$ with ${\mathbf{Y}}$, it also follows that

[TABLE]

This concludes the proof of Corollary 9. ∎

Remark 3**.**

Corollary 9 implies that the MMSE of estimating $\vec{\mathbf{X}}$ from an observation $\vec{\mathbf{Y}}$ grows sublinearly in $n$. This is significantly different from the MMSE of estimating ${\mathbf{X}}$ from an observation ${\mathbf{Y}}$, which grows linearly with $n$, namely [15]

[TABLE]

To prove Theorem 8, we next derive novel approximations on the first moment and variance of order statistics, which are tight when $n\to\infty$. We start by proposing the following lower bound on the variance, which will be used in Section VII.

Theorem 10**.**

For any $\vec{{\bf X}}$, we have that

[TABLE]

In particular, for ${\bf X}\sim\mathcal{N}({\bf 0}_{n},\mathbf{I}_{n})$ we have the following:

• •

For every $n\geq 1$

[TABLE]

where $\Gamma(\cdot)$ is the gamma function; and

• •

$n\to\mathsf{Var}(\|{\bf X}\|)$* is monotonically increasing with*

[TABLE]

Proof:

The proof of the lower bound in (28) follows by using modulus inequality

[TABLE]

The proofs of (29) and (30) follow from the fact that for ${\bf X}\sim\mathcal{N}({\bf 0}_{n},\mathbf{I}_{n})$, then $\|{\bf X}\|$ is distributed according to a chi distribution for which closed-form expressions and other properties of the variance can be found in [16, Chapter 8]. ∎

Next, we give an approximation for the variance that is accurate within $\sqrt{n\log(n)}$, the proof of which is in Appendix E.

Theorem 11**.**

Let ${\bf X}\sim\mathcal{N}({\bf 0}_{n},\mathbf{I}_{n})$. Then,

[TABLE]

and

[TABLE]

where $\Phi^{-1}(\cdot)$ is the inverse cumulative distribution function of the normal distribution (i.e., quantile function).

Remark 4**.**

Note that, by inspection, we have that

[TABLE]

where $U_{n}$ is an uniform random variable distributed on $\{\frac{1}{n+1},\ldots\frac{n}{n+1}\}$. This alternative expression can be used, for example, in the bound in (32).

To conclude the proof of Theorem 8, we require the following technical lemma whose proof can be found in Appendix G.

Lemma 12**.**

Let $U_{n}$ be uniformly distributed on $\{\frac{1}{n+1},\ldots\frac{n}{n+1}\}$ and let $U$ be uniform on $(0,1)$. Then,

[TABLE]

As a consequence of Theorem 11, Remark 4 and Lemma 12, we obtain

[TABLE]

where the equality in $\rm{(a)}$ follows since $X\stackrel{{\scriptstyle d}}{{=}}\Phi^{-1}(U)$. This concludes the proof of Theorem 8.

VII Discussion on Lower Bounds

In this section, we seek to further analyze the expression of $\mathrm{mmse}(\vec{\mathbf{X}}|\vec{\mathbf{Y}})$. Towards this end, we focus on lower bounds on the MMSE, with the goal to asses fundamental limits of recovering $\vec{\mathbf{X}}$ from $\vec{\mathbf{Y}}$. In particular, we discuss two commonly used lower bounds, namely the Bayesian Cramér-Rao bound and the maximum entropy bound. Surprisingly, and somewhat disappointingly, we show that both these commonly used lower bounds are unsuitable for our purposes. This opens up new research avenues of deriving tight lower bounds on $\mathrm{mmse}(\vec{\mathbf{X}}|\vec{\mathbf{Y}})$, which is object of current investigation.

VII-A Bayesian Cramér-Rao Bound

We here focus on the Bayesian Cramér-Rao bound. An important thing to note about the Bayesian Cramér-Rao is that this bound holds under the assumption of the following regularity condition [17]:

[TABLE]

where

[TABLE]

We next show that the regularity condition in (33) does not hold and, hence, the Bayesian Cramér-Rao bound cannot be used to lower bound $\mathrm{mmse}(\vec{\mathbf{X}}|\vec{\mathbf{Y}})$.

Lemma 13**.**

Let ${\bf X}\sim\mathcal{N}(\mathbf{0}_{n},\mathbf{I}_{n})$ and let ${\bf Y}|{\bf X}=\mathbf{x}\sim\mathcal{N}({\bf x},\sigma^{2}\mathbf{I}_{n})$ with $\sigma=1$. Then, for $n=2$ there exists an open $S\subset\vec{\mathbb{R}}_{n}$ such that

[TABLE]

Proof:

We start by noting that

[TABLE]

Therefore,

[TABLE]

where the labeled equalities follow from: ${\rm{(a)}}$ the definition of expectation and using (35); ${\rm{(b)}}$ using Bayes’ rule; ${\rm{(c)}}$ using Lemma 15 and Lemma 16; and ${\rm{(d)}}$ taking the gradient.

Now, since the above expression is an integral of Gaussian functions on $\vec{\mathbb{R}}_{n}$, it follows that $\vec{\mathbf{y}}\to\mathbb{E}\left[\phi(\vec{\mathbf{X}},\vec{\mathbf{Y}})|\vec{\mathbf{Y}}=\vec{\mathbf{y}}\right]$ is a continuous function on $\vec{\mathbb{R}}_{n}$. Thus, by continuity, if we can demonstrate that

[TABLE]

then

[TABLE]

in some open subset of $\vec{\mathbb{R}}_{n}$. This would violate the condition in (33). By setting $\vec{\mathbf{y}}=\mathbf{0}_{n}$, we have that

[TABLE]

where in the last step we have used that $f_{\vec{\mathbf{Y}}}(\mathbf{0}_{n})=\frac{n!}{(4\pi)^{\frac{n}{2}}}$ (see Lemma 15).

We now consider each of the $n=2$ coordinates of (38). Towards this end, we need to compute the following integral:

[TABLE]

for $i\in[2]$. Combining (38) with the integral above, we obtain

[TABLE]

The expression in (41) shows that (36) holds. This concludes the proof of Lemma 13. ∎

VII-B Maximum Entropy Bound

We here consider the maximum entropy bound on the MMSE [18, Chapter 2.2], which uses information theoretic arguments and states that for any pair of random vectors $(\mathbf{V},\mathbf{U})$ of dimension $n$, we have that (by also using the upper bound on the determinant of a matrix in terms of the trace)

[TABLE]

where $h(\mathbf{V}),h(\mathbf{V}|\mathbf{U})$ and $I(\mathbf{V};\mathbf{U})$ are the differential entropy, the conditional differential entropy and the mutual information, respectively. The inequality in (42) holds without any assumption on regularity conditions. We now use the inequality in (42) to derive the following lower bound.

Theorem 14**.**

Let ${\bf X}\sim\mathcal{N}(\mathbf{0}_{n},\mathbf{I}_{n})$ and let ${\bf Y}|{\bf X}=\mathbf{x}\sim\mathcal{N}({\bf x},\sigma^{2}\mathbf{I}_{n})$. Then,

[TABLE]

where

[TABLE]

Moreover, as $\sigma\to\infty$ the bound in (43) reduces to

[TABLE]

Proof:

Specializing the bound in (42) to our setting we have the following inequality:

[TABLE]

We first compute $h(\vec{\mathbf{X}})$ in (45), and we obtain

[TABLE]

where the labeled equalities follow from: ${\rm{(a)}}$ Lemma 15; ${\rm{(b)}}$ using that $f_{{\mathbf{X}}}(\mathbf{x})=\frac{1}{(2\pi)^{\frac{n}{2}}}\mathrm{e}^{-\frac{\|\mathbf{x}\|^{2}}{2}}$; and ${\rm{(c)}}$ using the fact that $\int_{\vec{\mathbb{R}}_{n}}f_{\vec{\mathbf{X}}}(\mathbf{x})\|\mathbf{x}\|^{2}{\rm d}\mathbf{x}=\mathbb{E}\left[\|\vec{\mathbf{X}}\|^{2}\right]=n$.

Next, we further upper bound $I(\vec{\mathbf{X}};\vec{\mathbf{Y}})$ in (45). This is done by using the data processing inequality in view of the Markov chain $\vec{\mathbf{X}}\to{\bf X}\to{\bf Y}\to\vec{\mathbf{Y}}$, that is

[TABLE]

where the equality follows from the closed-form expression for the mutual information of Gaussian random vectors [19, Chapter 9]. Thus, combing (45), (46) and (47) concludes the proof of (43) in Theorem 14.

We now prove (44) in Theorem 14. To this end, we take $\sigma\to\infty$ on both side of (43), which leads to

[TABLE]

where we have used that $\lim_{\sigma\to\infty}\mathrm{mmse}(\vec{\mathbf{X}}|\vec{\mathbf{Y}})=\mathsf{Var}(\vec{\mathbf{X}})$ (see Theorem 6) and the approximation is due to Stirling’s formula. This concludes the proof of Theorem 14. ∎

The expression in (43) is pleasing in its simplicity and, hence, it would be interesting to evaluate the effectiveness of this bound. In particular, it would be helpful to understand how the bound in (43) scales as $n$ tends to infinity. However, after closer examination, for $n\to\infty$ by comparing the lower bound in (44) with the lower bound on $\mathsf{Var}(\vec{\mathbf{X}})$ in (28), it is apparent that the bound in (44), which decreases with $n$, does not scale properly as $n$ tends to infinity.

Appendix A Auxiliary Results on the Distribution of $(\vec{\mathbf{X}},\vec{\mathbf{Y}})$

In this section, we state and prove some properties on exchangeable random variables.

Lemma 15**.**

Let $\mathbf{X}$ be an exchangeable random vector. Then,

[TABLE]

and

[TABLE]

Proof:

The proof of (48) follows by inspection.

To show (49) observe that

[TABLE]

where the labeled equalities follow from: $\rm{(a)}$ using (48); $\rm{(b)}$ the sifting property of the delta function; and $\rm{(c)}$ the exchangeable property of $\mathbf{X}$. This concludes the proof of Lemma 15. ∎

Lemma 16**.**

Let $\mathbf{X}$ be an exchangeable random vector (see Definition 1). Then,

[TABLE]

and

[TABLE]

Proof:

To show (50), observe that

[TABLE]

where the labeled equalities follow from: $\rm{(a)}$ using the law of total probability and the Markov chain $\vec{\mathbf{X}}\to{\bf X}\to\vec{\mathbf{Y}}$; $\rm{(b)}$ using Bayes’ rule; $\rm{(c)}$ using the law of total probability and the Markov chain ${\bf X}\to{\bf Y}\to\vec{\mathbf{Y}}$; $\rm{(d)}$ using the fact that (see (48) in Lemma 15)

[TABLE]

and the sifting property of the delta function; $\rm{(e)}$ using (48) in Lemma 15 and the sifting property of the delta function; $\rm{(f)}$ using the assumption of ${\bf X}$ being an exchangeable random vector; $\rm{(g)}$ using the assumption in (2); and $\rm{(h)}$ using (49) in Lemma 15. The proof of (51) now follows by using Bayes’ rule, the exchangeable property of ${\bf X}$ and (50); we obtain

[TABLE]

This concludes the proof of Lemma 16. ∎

Appendix B Proof of Theorem 2

Using (4), (7a) and modulus inequality, we have that

[TABLE]

where we let

[TABLE]

Next, observe that

[TABLE]

where the labeled (in)-equalities follow from: $\rm{(a)}$ using the property that $\mathbb{E}[Uf(W)|W]=\mathbb{E}[U|W]f(W)$; $\rm{(b)}$ using Cauchy-Schwarz inequality; and ${\rm{(c)}}$ using

[TABLE]

where we used the fact that, for an event $E$, we have $\mathbb{E}\left[1_{E}\right]=p(E)$ (property of indicator function).

Now, by using the Pythagorean theorem observe that

[TABLE]

Next, using the bound in (53) on each term in (54) we have the following bound:

[TABLE]

where the equality in $\rm{(a)}$ follows by using the property that $\mathbb{E}[Uf(W)|W]=\mathbb{E}[U|W]f(W)$. Combining (54) and (55) concludes the proof of Theorem 2.

Appendix C Proof of Lemma 3

From (11), we have that

[TABLE]

We now focus on the summation term in (57) and we obtain

[TABLE]

where the labeled equalities follow from: $\rm{(a)}$ Bayes’ theorem and since ${\bf Y}$ is exchangeable; $\rm{(b)}$ swapping the role of the sum and integration; $\rm{(c)}$ using (50) in Lemma 16 and (49) in Lemma 15; $\rm{(d)}$ using the definition of marginal PDF; and $\rm{(e)}$ using (49) in Lemma 15 and since ${\bf Y}$ is an exchangeable random variable. This concludes the proof of Lemma 3.

Appendix D Proof of Theorem 4

By using the Pythagorean theorem, we obtain (56) at the top of the next page, where the labeled (in)-equalities follow from: $\rm{(a)}$ using the optimal estimator in (4) and the suboptimal estimator in (12); $\rm{(b)}$ the fact that $\sum_{\pi\in\mathcal{P}}\mathbb{P}[{\bf X}\in\vec{\mathbb{R}}_{n}|\mathbf{Y}=\mathbf{P}_{\pi}\vec{\bf y}]=1$, as proved in Appendix C; $\rm{(c)}$ using Cauchy-Schwarz inequality; and $\rm{(d)}$ the fact that the probability is less than or equal to one.

We now show that (56) goes to zero as $t\to\infty$. We have

[TABLE]

where the labeled equalities follow from: ${\rm{(a)}}$ exchanging the expectation and the limit; this follows from the dominated convergence theorem by using the bound

[TABLE]

and $\rm{(b)}$ from the assumption in (14). This concludes the proof of Theorem 4.

Appendix E Proof of Theorem 11

E-A Proof of (31)

We have

[TABLE]

where the equalities follow from: $\rm{(a)}$ [3, eq.(2.3.7)]

[TABLE]

where $\mathbf{U}$ is standard uniform; moreover, $U_{(i)}$ is a beta $\beta(i,n-i+1)$ variate; $\rm{(b)}$ the fundamental theorem of calculus, i.e.,

[TABLE]

$\rm{(c)}$ linearity of expected value; $\rm{(d)}$ $U_{(i)}$ is a beta $\beta(i,n-i+1)$ variate, whose expected value is $\mathbb{E}[U_{(i)}]=\frac{i}{n+1}$.

Thus, from (58) we obtain

[TABLE]

Therefore, we now focus on upper bounding $\left|\mathbb{E}[R_{0}(U_{(i)})]\right|$. First, let $f(\cdot)$ be the PDF of the standard Gaussian random variable, and note that

[TABLE]

where in the last step, we have used the property

[TABLE]

and the change of variable $x=tU_{(i)}+(1-t)\mathbb{E}[U_{(i)}]$. Thus,

[TABLE]

where $\rm{(e)}$ follows by using Jensen’s inequality. We now focus on upper bounding the integral term. We have

[TABLE]

where the equalities/inequalities follow from: $\rm{(f)}$ by letting $x(t)=tU_{(i)}+(1-t)\mathbb{E}[U_{(i)}]$, and since

[TABLE]

which follows from the fact that $f\left(\Phi^{-1}(u)\right)$ is a concave and symmetric function [20, Lemma 10.1]; and $\rm{(g)}$ the fact that $\min\{a,b\}\geq\frac{ab}{a+b}.$ Therefore, we obtain

[TABLE]

where ${\rm{(h)}}$ follows from Jensen’s inequality. Now, observe that

[TABLE]

This follows from the property $\mathbb{E}\left[\left|X-c\right|^{2}\right]=\mathsf{Var}[X]+\left(\mathbb{E}[X]-c\right)^{2},$ where $c$ is a constant. Moreover, for $U_{(i)}$, i.e., a beta $\beta(i,n-i+1)$ variate, we have [21]

[TABLE]

where $\psi_{1}(\cdot)$ is the second of the polygamma functions, and $\psi(\cdot)$ is the digamma function. Thus, with this we obtain

[TABLE]

where $\ell=\log\left(\frac{i}{n+1-i}\right).$ Finally, observe that [22]

[TABLE]

With this, we obtain

[TABLE]

This concludes the proof of (31).

E-B Proof of (32)

We have that

[TABLE]

where the labeled (in)-equalities follow from: $\rm{(a)}$ the fact that $\mathsf{Var}(\vec{\mathbf{X}})=n-\sum_{i=1}^{n}\mathbb{E}^{2}[X_{(i)}]$; ${\rm{(b)}}$ defining

[TABLE]

and $\rm{(c)}$ using Cauchy-Schwarz inequality.

We now bound each term in (61). Towards this end, we use the following lemma, whose proof is provided in Appendix F.

Lemma 17**.**

For any $n$ and any $\epsilon\geq 0$111Note that the case $\epsilon=4$ should be computed by taking the limit.

[TABLE]

By using Lemma 17, we have the following bound

[TABLE]

Then, by using (31), we obtain

[TABLE]

where the labeled inequalities follow from: $\rm{(d)}$ for $a,b$ such that $\min\{a,b\}>0$, we have $|a-b|^{2}=(a-b)^{2}\leq a^{2}+b^{2}$; and $\rm{(e)}$ using the integral approximation lemma, namely

[TABLE]

where $f(\cdot)$ is a monotone decreasing function.

Now, by collecting (61), (62) and (63) we have that

[TABLE]

This concludes the proof of (32), and hence of of Theorem 11.

Appendix F Proof of Lemma 17

First, observe that from (59), we have that

[TABLE]

where $f(\cdot)$ is the PDF of the standard Gaussian random variable. Now, we have that

[TABLE]

where the labeled (in)-equalities follow from: $\rm{(a)}$ the symmetry of $\left|\Phi^{-1}\left(\frac{i}{n+1}\right)\right|$; $\rm{(b)}$ using (65) and (60) by noting that $i\leq\frac{n+1}{2}$; ${\rm{(c)}}$ using the upper bound $\log(x)\leq\frac{x^{s}}{s},\ \forall s>0,$ with $s=1/4$; and $\rm{(d)}$ using the integral approximation in (64). This concludes the proof of Lemma 17.

Appendix G Proof of Lemma 12

Let $V_{n}^{2}=\left|\Phi^{-1}(U_{n})\right|^{2}$. It is not difficult to check that $U_{n}\to U$ in distribution [23, Example 8.2.1] and, therefore, by the continuity of $x\to\left|\Phi^{-1}(x)\right|^{2}$, we have that

[TABLE]

in distribution. To conclude the proof it remains to show that

[TABLE]

This exchange of integration and expectation under convergence in distribution is guaranteed by the Vitaly convergence theorem [24, Chapter 6], provided that the sequence $\{V_{n}^{2}\}_{n=1}^{\infty}$ is uniformly integrable.

To show that $\{V_{n}^{2}\}_{n=1}^{\infty}$ is uniformly integrable, we check the crystal ball condition [23, Chapter 6.5.1] given by

[TABLE]

for some $\delta>0$. To see this, we choose $\delta=2$ and we observe that, in view of Lemma 17 with $\epsilon=2$, we obtain

[TABLE]

which is clearly bounded.

Therefore, $V_{n}^{2}$ is uniformly integrable and the interchange of the expectation and limit in (66) is verified. This concludes the proof of Lemma 12.