SLOPE for Sparse Linear Regression:Asymptotics and Optimal Regularization

TL;DR

This paper provides a detailed asymptotic analysis of the SLOPE estimator in sparse linear regression, establishing fundamental limits and optimal regularization design, supported by simulations.

Contribution

It introduces a precise asymptotic performance characterization of SLOPE and develops a method to optimally design regularization sequences for improved estimation and variable selection.

Findings

Optimal regularization sequences outperform existing designs.

Asymptotic limits for estimation and variable selection are established.

Efficient finite-dimensional approximations enable practical implementation.

Abstract

In sparse linear regression, the SLOPE estimator generalizes LASSO by penalizing different coordinates of the estimate according to their magnitudes. In this paper, we present a precise performance characterization of SLOPE in the asymptotic regime where the number of unknown parameters grows in proportion to the number of observations. Our asymptotic characterization enables us to derive the fundamental limits of SLOPE in both estimation and variable selection settings. We also provide a computational feasible way to optimally design the regularizing sequences such that the fundamental limits are reached. In both settings, we show that the optimal design problem can be formulated as certain infinite-dimensional convex optimization problems, which have efficient and accurate finite-dimensional approximations. Numerical simulations verify all our asymptotic predictions. They demonstrate the superiority of our optimal regularizing sequences over other designs used in the existing literature.

Equations

$$\mathbf{y}=\mathbf{A}\mathbf{\beta }+\mathbf{w},$$

$$\hat{\mathbf{\beta }}\in \underset{\mathbf{b}}{\mathrm{argmin}}\frac{1}{2}\Vert \mathbf{y}-\mathbf{A}\mathbf{b}{\Vert }_2^2+\sum _{i=1}^p{\lambda }_i|b{|}_{(i)},$$

$${\text{Prox}}_{\mathbf{\lambda }}(\mathbf{y})\stackrel{\text{def}}{=}\underset{\mathbf{x}}{\mathrm{argmin}}\frac{1}{2}\Vert \mathbf{y}-\mathbf{x}{\Vert }_2^2+J_{\mathbf{\lambda }}(\mathbf{x})$$

$$\frac{1}{p}\Vert {\text{Prox}}_{\mathbf{\lambda }}(\mathbf{y})-\eta (\mathbf{y}){\Vert }^2\to 0,$$

$$\hat{\mathbf{\beta }}\in \underset{\mathbf{b}}{\mathrm{argmin}}\sum _{i=1}^n\ell (y_i,{\mathbf{a}}_i^{\top }\mathbf{b})+r(\mathbf{b})$$

$$W_{q}(\mu,\nu)\overset{\text{def}}{=}\Big{(}\inf_{\pi\in\Pi(\mu,\nu)}\mathbb{E}\|X-Y\|_{2}^{q}\Big{)}^{1/q},$$

$${\mathcal{M}}_{\mathbf{\lambda }}(\mathbf{y};\tau )\stackrel{\text{def}}{=}\underset{\mathbf{x}}{\min }\frac{1}{2\tau }\Vert \mathbf{y}-\mathbf{x}{\Vert }_2^2+J_{\mathbf{\lambda }}(\mathbf{x}),$$

$${\mathcal{M}}_{{\mu }_{\Lambda }}({\mu }_Y;\tau )\stackrel{\text{def}}{=}\underset{g\in \mathcal{I}}{\min }\frac{1}{2\tau }{\mathbb{E}}_{{\mu }_Y}[Y-g(Y){]}^2+{\int }_0^1{F}_{\Lambda }^{-1}(u)F_{|g(Y)|}^{-1}(u)du,$$

$$\mathcal{I}\stackrel{\text{def}}{=}\{g(y)\mid g(y)\text{ is odd, non-decreasing and 1-Lipschitz}\}.$$

$$\frac{1}{p}{\mathcal{M}}_{\mathbf{\lambda }}(\mathbf{y};\tau )\to {\mathcal{M}}_{{\mu }_{\Lambda }}({\mu }_Y;\tau )$$

$$\frac{1}{p}\Vert {\text{Prox}}_{\tau \mathbf{\lambda }}(\mathbf{y})-\eta (\mathbf{y};{\mu }_Y,{\mu }_{\tau \Lambda }){\Vert }^2\to 0,$$

$$\displaystyle\min_{g\in\mathcal{I}}\frac{1}{2}\mathbb{E}\big{[}\underbrace{|Y|-\lambda}_{:=f(|Y|)}-g(|Y|)\big{]}^{2}+\text{constant}.$$

$$\eta (y;{\mu }_Y,{\mu }_{\Lambda })$$

$$=\text{sign}(y)\max (|y|-\lambda ,0),$$

$${\int }_0^1{F}_{\Lambda }^{-1}(u)F_{|g(Y)|}^{-1}(u)du$$

$$\displaystyle\min_{g\in\mathcal{I}}\frac{1}{2}\mathbb{E}\big{[}|Y|-g(|Y|)\big{]}^{2}+\mathbb{E}\big{[}\lambda(|Y|)g(|Y|)\big{]}$$

$$=$$

$$\frac{1}{p}\sum _{i=1}^p\psi ({\hat{\beta }}_i,{\beta }_i)\stackrel{\mathbb{P}}{\to }\mathbb{E}[\psi (\eta (Y_{\ast };{\mu }_{Y_{\ast }},{\mu }_{{\tau }_{\ast }\Lambda }),B)],$$

$${\sigma }^2$$

$$1$$

$$\underset{p\to \infty }{\lim }\frac{1}{p}\Vert \hat{\mathbf{\beta }}-\mathbf{\beta }{\Vert }^2=\delta ({\sigma }_{\ast }^2-{\sigma }_w).$$

$$\min_{\sigma\geq\sigma_{w}}\max_{\theta\geq 0}\frac{\theta}{2}\Big{(}\frac{\sigma_{w}^{2}}{\sigma}+\sigma\Big{)}-\frac{\theta^{2}}{2}+\frac{1}{\delta}\Big{[}\underbrace{\lim_{p\to\infty}\frac{1}{p}\mathcal{M}_{\boldsymbol{\lambda}}\big{(}\boldsymbol{\beta}+\sigma\boldsymbol{h};\frac{\sigma}{\theta}\big{)}}_{:=\mathcal{F}(\sigma,\theta)}-\frac{\theta\sigma}{2}\Big{]},$$

$$R_0^{(p)}=$$

$$\text{Type-I error}=$$

$${\Lambda }_{\vartheta }=\{\begin{array}{ll}\vartheta & |Y_{\ast }|<\vartheta {\tau }_{\ast },\\ \frac{|Y_{\ast }|}{{\tau }_{\ast }}& \vartheta {\tau }_{\ast }\le |Y_{\ast }|<{\tau }_{\ast },\\ 1& |Y_{\ast }|\ge {\tau }_{\ast },\end{array}$$

$$\displaystyle R_{0}^{(p)}\overset{\mathbb{P}}{\rightarrow}\mathbb{P}\big{(}\eta(Y_{*})=0\big{)}\text{ and }V^{(p)}\overset{\mathbb{P}}{\rightarrow}\mathbb{P}\big{(}\eta(Y_{*})\neq 0,B=0\big{)},$$

$$\underset{p\to \infty }{\lim }\text{Type-I error}$$

$$\underset{p\to \infty }{\lim }\text{Power}=$$

$$\underset{{\mu }_{\Lambda }\in {\mathcal{P}}_{\Lambda }}{\inf }\underset{p\to \infty }{\lim }\frac{1}{p}\Vert \hat{\mathbf{\beta }}-\mathbf{\beta }{\Vert }_2^2$$

$$\underset{{\mu }_{\Lambda }\in {\mathcal{P}}_{\Lambda }}{\inf }{\sigma }_{\ast }.$$

$${\sigma }^2$$

$$1$$

Full text

I Introduction

I-A Motivation and Problem Setup

In sparse linear regression, we seek to estimate a sparse vector $\boldsymbol{\beta}\in\mathbb{R}^{p}$ from

[TABLE]

where $\boldsymbol{A}\in\mathbb{R}^{n\times p}$ is the design matrix and $\boldsymbol{w}$ denotes the observation noise. In this paper, we study the sorted $\ell_{1}$ penalization estimator (SLOPE) [2] (see also [3, 4]), a new paradigm for sparse linear regression. Given a non-decreasing regularization sequence $\boldsymbol{\lambda}=[\lambda_{1},\lambda_{2},\ldots,\lambda_{p}]^{\top}$ with $0\leq\lambda_{1}\leq\lambda_{2}\leq\cdots\leq\lambda_{p}$, SLOPE estimates $\boldsymbol{\beta}$ by solving the following optimization problem

[TABLE]

where $|b|_{(1)}\leq|b|_{(2)}\leq\cdots\leq|b|_{(p)}$ is a reordering of the absolute values $\left\lvert b_{1}\right\rvert,\left\lvert b_{2}\right\rvert,\ldots,\left\lvert b_{p}\right\rvert$ in increasing order. In [2], the regularization term $J_{\boldsymbol{\lambda}}(\boldsymbol{b})\overset{\text{def}}{=}\sum_{i=1}^{p}\lambda_{i}|b|_{(i)}$ is referred to as the “sorted $\ell_{1}$ norm” of $\boldsymbol{b}$. The same regularizer was independently developed in a different line of work [5, 3, 4, 6], where the motivation is to promote group selection in the presence of correlated covariates.

The classical LASSO estimator is a special case of SLOPE. It corresponds to using a constant regularization sequence, i.e., $\lambda_{1}=\lambda_{2}=\cdots=\lambda_{p}=\lambda$. However, with more general $\boldsymbol{\lambda}$-sequences, SLOPE has the flexibility to penalize different coordinates of the estimate according to their magnitudes. This adaptivity endows SLOPE with some nice statistical properties that are not possessed by LASSO. For example, it is shown in [7, 8] that SLOPE achieves the minimax $\ell_{2}$ estimation rate with high probability. When applied in variable selection problem, SLOPE is shown to control the false discovery rate (FDR) for orthogonal design matrices [2], which is not the case for LASSO. In addition, the new regularizer $J_{\boldsymbol{\lambda}}(\boldsymbol{b})$ is still a norm [2, 4]. Thus, the optimization problem associated with SLOPE remains convex, and it can be efficiently solved by using e.g., proximal gradient descent [4, 2].

Although the flexible regularization of SLOPE creates the hope of potential performance enhancement, to fully realize SLOPE’s potential, we have to carefully design the regularizing sequence $\boldsymbol{\lambda}$. Note that this is equivalent to specifying the empirical distribution of $\boldsymbol{\lambda}$. Popular choices in the previous works include delta distribution (i.e., LASSO), uniform distribution [5], chi-distribution [9], etc. These regularization schemes are mostly devised based on statistical insights gained from simpler models and they are indeed superior than LASSO in several applications. However, the success of these regularizing sequences provide no quantitative answer to the following two questions:

What is the fundamental limit of SLOPE? 2. 2.

How to optimally design $\boldsymbol{\lambda}$ to reach the fundamental limit?

The aforementioned studies on analyzing SLOPE provide very limited information for us to address the above two questions, since in these works, the SLOPE’s performance is characterized in an order-wise manner, which contains loose constants. What we need is an exact performance characterization of SLOPE estimator, which is still absent in the existing literature. On the other hand, however, exact asymptotic analysis has been carried out for LASSO [10, 11] and several other regularized regression techniques [12, 13, 14, 15], under certain statistical assumptions on the sensing matrix $\boldsymbol{A}$. One key feature of all these results is that the performance in the originally high-dimensional model can be well-captured by some low dimensional problems, which are much easier to handle. The technical hurdle that has precluded a similar treatment for SLOPE is that unlike all the regularizer considered in these works, the SLOPE norm $J_{\boldsymbol{\lambda}}(\boldsymbol{x})$ is non-separable: it cannot be written as a sum of component-wise functions, i.e., $J_{\boldsymbol{\lambda}}(\boldsymbol{x})\neq\sum_{i=1}^{p}J_{i}(x_{i})$. This makes a similar low-dimensional reduction more challenging.

I-B Main Contributions

In this paper, we answer the questions raised above. Our main contributions are listed as follows:

I-B1 Asymptotic separability

As mentioned above, the main obstacle in analyzing SLOPE asymptotics is the non-separability of SLOPE regularizer $J_{\boldsymbol{\lambda}}(\boldsymbol{b})=\sum_{i=1}^{p}\lambda_{i}|b|_{(i)}$. We overcome this challenge by showing that the proximal operator of $J_{\boldsymbol{\lambda}}(\boldsymbol{b})$ is asymptotically separable. To be more concrete, we first give a technically light overview of this result. The proximal operator of $J_{\boldsymbol{\lambda}}(\boldsymbol{b})$ is defined as:

[TABLE]

In the case of LASSO, where we choose $\lambda_{1}=\lambda_{2}=\cdots=\lambda_{p}=\lambda$, characterizing $\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{y})$ is easy, since the optimization in (3) is equivalent to $p$ scalar problems: $\sum_{i=1}^{p}\min_{x_{i}}\frac{1}{2}(y_{i}-x_{i})^{2}+\lambda|x_{i}|.$ Correspondingly, the proximal operator is separable: $[\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{y})]_{i}=\text{sign}(y_{i})\max(|y_{i}|-\lambda,0).$ In other words, the $i$th element of $\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{y})$ is solely determined by $y_{i}$. However, this separability property does not hold for a general regularizing sequence. When $p$ is finite, $[\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{y})]_{i}$ depends not only on $y_{i}$ but also on other elements of $\boldsymbol{y}$. As one of the core results in this paper, we show that if the empirical distributions of $\boldsymbol{y}$ and $\boldsymbol{\lambda}$ converge as $p\to\infty$, then

[TABLE]

where $\eta$ is a limiting scalar function that is uniquely determined by the limiting empirical measures of $\boldsymbol{y}$ and $\boldsymbol{\lambda}$ (for the exact form, see Proposition 1). This result is illustrated in Fig. 1, where we compare the actual proximal operator $\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{y})$ and the limiting scalar function $\eta(y)$, for two different $\boldsymbol{\lambda}$-sequences shown in Fig. 1a and Fig. 1c. It can be seen that under a moderate dimension, the proximal operator $\text{Prox}_{{\tau\boldsymbol{\lambda}}}(\boldsymbol{y})$ can already be very accurately approximated by $\eta(\boldsymbol{y})$.

I-B2 Exact characterization

The asymptotic separability allows us to obtain the exact characterization of SLOPE’s performance in the linear asymptotic regime: $n,\,p\to\infty$ and $n/p\to\delta$, under the assumption that sensing matrix $\boldsymbol{A}$ is generated from i.i.d. Gaussian. On a high level, our main results show that the joint empirical distribution of $\{(\hat{\beta}_{i},\beta_{i})\}_{i=1}^{p}$ converges to a well-defined limiting measure (the precise description can be found in Theorem 1). Note that the performance metrics of interests such as mean square error (MSE), type-I error, power are all functional of the empirical measure $\{(\hat{\beta}_{i},\beta_{i})\}_{i=1}^{p}$. Therefore, this makes it possible us to compute the high-dimensional limits of all these quantities. Compared with the probabilistic bounds derived in previous work, our results are asymptotically exact.

I-B3 Fundamental limits and optimal regularizagion

The exact asymptotic characterization finally enables us to derive the fundamental limits of SLOPE in both estimation and variable selection tasks: (1) the minimum MSE that can be achieved by SLOPE; and (2) the highest possible power achievable under any given level of Type-I error. Moreover, we show that in both cases, the optimal $\boldsymbol{\lambda}$ sequence can be obtained by solving certain infinite-dimensional convex optimization problems, which have efficient and accurate finite-dimensional approximations. It is worth mentioning that a caveat of our optimal design is that it requires knowing the limiting empirical measure of $\boldsymbol{\beta}$ (e.g., the sparsity level and the distribution of its nonzero coefficients). For this reason, our results are oracle optimal. However, it provides the first step towards optimal sequence designs under more realistic setting, where no or only limited information about $\boldsymbol{\beta}$ is available.

An illustration of asymptotic characterization and optimality results stated above are presented in Fig. 2. We consider three different regularizing sequences: LASSO, BHq sequence proposed in [9] and the optimal sequence given by Proposition 4 below. In Fig. 2a, we plot the empirical MSEs and compare them with the theoretical results. We can see they match well under all settings. Moreover, all the recorded MSE values are lower bounded by the fundamental limits predicted by our theory (red curve in the figure) and they can be achieved by the optimally designed sequences (red circles in the figure). For comparison, we also enclose the curve of minimum mean square error (MMSE) of linear Gaussian model, which was derived in [16, 17]. Finally, to help the readers get a sense of what the optimal regularizing sequences look like, in Fig. 2b-2d we plot their empirical distributions under 3 different sampling ratios $\delta$. Interestingly, we can find they exhibit very different distributions as we change $\delta$.

I-C Related Work

I-C1 Exact asymptotic characterization

There has been a growing body of works studying the exact asymptotics in high-dimensional statistical problems under random design assumptions. A partial list of these works includes [18, 19, 20, 21, 10, 22, 23, 24, 25, 26, 27, 28, 29, 30]. One distinct feature of these type of results is that they provide sharp performance guarantee that does not contain loose constants. From a technical viewpoint, these works are built on powerful tools including statistical physics [31, 32], approximate message passing (AMP) [19, 20], Gaussian width or statistical dimensions [21, 25], leave-one-out analysis [24, 13], Gordon’s Gaussian comparison lemma [33], etc. Our main asymptotic characterization is proved based on convex min-max Gaussian theorem (CGMT) [34, 12, 26], which is a tight version of Gordon’s comparison lemma in the convex setting. This framework was developed through a series of works [34, 12, 26, 29] and have now been successfully applied in a variety of problems such as binary detection [14], regularized M-estimation [26, 29], phase retrieval [28, 35] and high-dimensional classification [36, 37, 38].

I-C2 Optimal M estimation in high dimensions

The optimality part of this work falls within the line of research pursuing the optimal M-estimation in high-dimensional regression. The general form of M-estimator is as follows:

[TABLE]

and the question is what is the optimal statistical performance achievable by (4) and how to optimally design the loss function $\ell$ and the regularizer $r$. The exact asymptotic characterizations open up the possibility of obtaining a precise answer to the above question. This line of research is initiated by the papers [39] and [27], where the authors study the fundamental limits of the unregularized M-estimator (i.e., the case when $r=0$) in the linear model. In particular, a computational feasible recipe is provided in [39] for constructing the optimal loss function $\ell$ that minimizes the estimation errors. Similar types of results are also recently established for the binary models [40]. When a regularizer is included, the optimal performance of (4) in the linear model is studied in [41] and recently extended to binary model for the special case of quadratic regularization [42, 43]. In the meantime, a series of papers study the optimal $\ell_{q}$-norm regularized least square regression [15, 44, 45]. In some limiting regimes, explicit answers are provided regarding the optimal choice of $q$. Note that all the aforementioned works consider the separable regularizer: $r(\boldsymbol{b})=\sum_{i=1}r_{i}(b_{i})$, while SLOPE regularizer considered in this paper is not separable.

Closely related with current work is the paper by Celentano and Montanari [46]. One of their main results is on the optimal estimation performance achievable by quadratic loss regularized by any lower semi-continuous, proper, convex and symmetric 111Symmetric means $r(\boldsymbol{b})$ is permutation invariant to coordinates of $\boldsymbol{b}$. function. It is not hard to check that SLOPE norm belongs to this family of functions. In fact, the optimality results presented in their paper and ours share a very similar form. We will elaborate more on this in Sec. IV-A.

I-C3 Three Parallel works

Finally, we mention three parallel works that also study the limiting behavior of SLOPE under the same asymptotic setting.

From an algorithmic perspective, [47] consider solving the SLOPE minimization problem (2) using the AMP algorithm. By relating the stationary point of AMP iterations to SLOPE estimator, they also establish the same characterization (as shown in Theorem 1 below). In the proof, they also utilize the asymptotic separability property proved in Proposition 1. 2. 2.

The CGMT framework is also applied in [48] to obtain the limiting mean square errors (MSE) of SLOPE, together with a finite-sample concentration bound. The authors quantitatively compares the MSEs of different regularizing sequences in some limiting regimes. In particular, it is shown that in the high SNR regimes, LASSO regularization is optimal. A major difference from our work is that they do not exploit the asymptotic separability of SLOPE and the optimal performance in the general regime is not addressed. 3. 3.

In [49], the asymptotic separability properties is further extended to all lsc, proper, convex and symmetric regularizers using an elegant lifting and embedding idea. A finite-sample concentration bound is also given. Using the general asymptotic separability results, the author proves a conjecture in [46]: the MSE lower bound achievable by non-separable convex symmetric regularizers will be the same if we are restricted to the separable convex regularizers. However, the performance of variable selection is not addressed.

I-D Notations

For a vector $\boldsymbol{x}\in\mathbb{R}^{p}$ and a scalar function $f(\cdot):\mathbb{R}\to\mathbb{R}$, $f(\boldsymbol{x})$ means $f(\cdot)$ is applied to vector $\boldsymbol{x}$ coordinate-wise. $\|\boldsymbol{x}\|$ denotes the $\ell_{2}$ norm, $x_{i}$ (or $[\boldsymbol{x}]_{i}$) denotes the $i$th coordinate of $\boldsymbol{x}$ and $|x|_{(i)}$ (or $|\boldsymbol{x}|_{(i)}$) denotes the $i$th largest coordinate of $|\boldsymbol{x}|$. The Euclidean ball in $\mathbb{R}^{p}$ centered on $\boldsymbol{a}$ with radius $r\geq 0$ is denoted as: $\mathcal{B}_{r}(\boldsymbol{a}):=\{\boldsymbol{v}:\|\boldsymbol{v}-\boldsymbol{a}\|\leq r\}$ and $\mathcal{B}_{r}\overset{\text{def}}{=}\mathcal{B}_{r}(\boldsymbol{0})$. Also we define $\mathcal{B}^{o}_{r}(\boldsymbol{a})\overset{\text{def}}{=}\{\boldsymbol{v}:\|\boldsymbol{v}-\boldsymbol{a}\|\geq r\}$.

For a probability measure $\mu$, we denote $\text{Supp}(\mu)$ as its support. For random variables $X,Y$, we denote $\mu_{X,Y}$ and $\mu_{X}$, $\mu_{Y}$ as their joint and marginal measures and $F_{X}$, $F_{Y}$ as the corresponding (marginal) cumulative distribution function (CDF). The quantile function of random variable $X$ is denoted as $F_{X}^{-1}(p)$, where $F_{X}^{-1}(p)\overset{\text{def}}{=}\inf\{x\in\mathbb{R}:F_{X}(x)\geq p\}$. Specifically, we use $\Phi$ and $\Phi^{-1}$ to denote the CDF and quantile function of standard Gaussian. For vectors $\boldsymbol{x},\boldsymbol{y}\in\mathbb{R}^{p}$, we denote $\mu_{\boldsymbol{x},\boldsymbol{y}}$ and $\mu_{\boldsymbol{x}}$, $\mu_{\boldsymbol{y}}$ as their joint and marginal empirical measures and $F_{\boldsymbol{x}}$, $F_{\boldsymbol{y}}$ as the corresponding (marginal) empirical CDF. Also we denote the empirical quantile function of $\boldsymbol{x}$ as $F_{\boldsymbol{x}}^{-1}$.

We denote $\mathcal{P}_{q}(\mathbb{R}^{k})$, for some $q\geq 1$ and $k\in\mathbb{Z}^{+}$, as the space of all probability measures on $\mathbb{R}^{k}$ with bounded moments of order $q$, i.e., for any $\mu\in\mathcal{P}_{q}(\mathbb{R}^{k})$, it holds that $\mathbb{E}_{\mu}(\|X\|^{q})<\infty$. For two measures $\mu,\nu\in\mathcal{P}_{q}(\mathbb{R}^{k})$, their Wasserstein-$q$ distance is defined as:

[TABLE]

where $(X,Y)\sim\pi$ and $\Pi(\mu,\nu)$ is the set of all couplings of $\mu$ and $\nu$.

I-E Asymptotic Setting

There are four main objects in the description of our model and algorithm: (1) the unknown vector $\boldsymbol{\beta}$; (2) the design matrix $\boldsymbol{A}$; (3) the noise vector $\boldsymbol{w}$; and (4) the regularizing sequence $\boldsymbol{\lambda}$. Since we study the asymptotic limit (with $p\to\infty$), we will consider a sequence of instances \big{\{}\boldsymbol{\beta}^{(p)},\,\boldsymbol{A}^{(p)},\,\boldsymbol{w}^{(p)},\,\boldsymbol{\lambda}^{(p)}\big{\}}_{p\in\mathbb{N}} with increasing dimensions $p$, where $\boldsymbol{\beta}^{(p)},\,\boldsymbol{\lambda}^{(p)}\in\mathbb{R}^{p}$, $\boldsymbol{A}^{(p)}\in\mathbb{R}^{n\times p}$ and $\boldsymbol{w}^{(p)}\in\mathbb{R}^{n}$. A sequence of vectors $\{\boldsymbol{x}^{(p)}\}_{p\in\mathbb{Z}}$ (or $\{\boldsymbol{x}^{(p)},\boldsymbol{y}^{(p)}\}_{p\in\mathbb{Z}}$), with $p$ indexing the growing dimensions, is called a converging sequence, if its empirical measure $\mu_{\boldsymbol{x}^{(p)}}$ (or $\mu_{\boldsymbol{x}^{(p)},\boldsymbol{y}^{(p)}}$) converges in Wasserstein-2 distance to a probability measure $\mu_{X}$ (or $\mu_{X,Y}$) as $p\to\infty$. For notational brevity, we will omit the superscript “$(p)$” when it is clear from the context.

I-F Paper Outline

The rest of the paper is organized as follows. In Sec. II, we first prove the asymptotic separability of the proximal operator associated with $J_{\boldsymbol{\lambda}}(\boldsymbol{x})$. This property allows us to derive our asymptotic characterization of SLOPE in Sec. III. Based on this analysis, we derive the fundamental limit and present the optimal design of the regularizing sequence in Sec. IV. Numerical simulations are provided to verify our asymptotic characterizations. They also demonstrate the superiority of our optimal regularization over LASSO and BHq sequence in [7]. In Sec. V, we provide the proof of all our main results. We conclude the paper in Sec. VI and discuss some possible directions for future work.

II Proximal Problem and Asymptotic Separability

We start by studying the following proximal problem:

[TABLE]

where $\tau>0$, $\boldsymbol{y}\in\mathbb{R}^{p}$ and $J_{\boldsymbol{\lambda}}(\boldsymbol{x})=\sum_{i=1}^{p}\lambda_{i}|x|_{(i)}$, with $0\leq\lambda_{1}\leq\lambda_{2}\leq\cdots\leq\lambda_{p}$. $\mathcal{M}_{\boldsymbol{\lambda}}(\boldsymbol{y};\tau)$ in (5) is known as the Moreau envelope of $J_{\boldsymbol{\lambda}}(\boldsymbol{x})$ evaluated at $\boldsymbol{y}$ and $\tau$ is the smoothing parameter. The unique minimizer of (5) is the proximal operator associated with $J_{\boldsymbol{\lambda}}(\boldsymbol{x})$ under parameter $\tau$. From (5), we know the proximal operator of $J_{\boldsymbol{\lambda}}(\boldsymbol{x})$ is fully determined by $\boldsymbol{y}$ and $\tau\boldsymbol{\lambda}$, so we simply denote it as $\text{Prox}_{\tau\boldsymbol{\lambda}}(\boldsymbol{y})$. It turns out that the asymptotics of the original problem (2) is closely related to (5). Thus, as a preliminary step, we will first analyze its limiting properties.

To state our result, we introduce the following functional optimization problem. For $\mu_{Y},\mu_{\Lambda}\in\mathcal{P}_{2}(\mathbb{R})$, with $\mathbb{P}(\Lambda\geq 0)=1$, define

[TABLE]

where

[TABLE]

Also we denote $\eta(\cdot;\mu_{Y},\mu_{\tau\Lambda})$ as the optimal solution of (6). Comparing (6) with (5), we can intuitively interpret $\mathcal{M}_{\mu_{\Lambda}}(\mu_{Y};\tau)$ and $\eta(\cdot;\mu_{Y},\mu_{\tau\Lambda})$ as the functional-form Moreau envelope and proximal operator.

We are now ready to state our main result on the asymptotics of the proximal problem (5).

Proposition 1

Let $\{\boldsymbol{y}\}_{p\in\mathbb{N}}$ and $\{\boldsymbol{\lambda}\}_{p\in\mathbb{N}}$ be two converging sequences, with limiting measures $\mu_{Y}$ and $\mu_{\Lambda}$ satisfying $\mathbb{P}(\Lambda\geq 0)=1$. It holds that for any $\tau>0$,

[TABLE]

and

[TABLE]

*where $\mathcal{M}_{\mu_{\Lambda}}(\mu_{Y};\tau)$ and $\eta(\cdot;\mu_{Y},\mu_{\tau\Lambda})$ are the optimal value and the unique (up to a set of measure zero with respect to $\mu_{Y}$) optimal solution of (6). *

The proof of Proposition 1 will be provided in Appendix V-A. We will also see that the limiting characterization of $\mathcal{M}_{\boldsymbol{\lambda}}(\boldsymbol{y};\tau)$ in (6) and the asymptotic separability of $\text{Prox}_{\tau\boldsymbol{\lambda}}(\cdot)$ in (9) greatly facilitates our asymptotic analysis and the optimal design of $\boldsymbol{\lambda}$, since this allows us to reduce the original high-dimensional problem to an equivalent one-dimensional problem, as in the LASSO case. Indeed, $\eta(\cdot;\mu_{Y},\mu_{\tau\Lambda})$ in (9) is exactly the limiting scalar function $\eta(\cdot)$ shown earlier in Fig. 1. We will still sometimes adopt the lighter notation $\eta(\cdot)$, when doing so causes no confusion.

Note that (6) is involved with an infinite-dimensional optimization, which typically permits no simple analytical solutions. To gain more intuition, before moving on, let us consider two examples where closed-form solutions do exist.

Example 1** (LASSO)**

The LASSO case corresponds to $\mathbb{P}(\Lambda=\lambda)=1$. When $\tau=1$, optimization in (6) then reduces to

[TABLE]

Note that the function $f(y)=y-\lambda$ in (10) is non-decreasing and 1-Lipschitz on $\mathbb{R}_{\geq 0}$ and $f(0)\leq 0$. It is not hard to show in this case, the optimal solution of (10) equals to

[TABLE]

which is exactly the soft-thresholding function.

Example 2** (BHq [9])**

The BHq regularization corresponds to $\Lambda\sim\Phi^{-1}(1-\frac{q}{2}+\frac{q}{2}U)$, where $q\in(0,1]$ and $U$ is uniformly distributed over $[0,1]$. Then we have $F_{\Lambda}^{-1}(u)=\Phi^{-1}(1-\frac{q}{2}+\frac{q}{2}u)$. Further, we consider $Y\sim\mathcal{N}(0,1)$. It holds that $F_{|Y|}(y)=2\Phi(y)-1$ and F_{|g(Y)|}^{-1}\big{(}F_{|Y|}(y)\big{)}=g(y), for $y\geq 0$. Therefore,

[TABLE]

where we apply a change of variable $u=F_{|Y|}(y)$. In this case, (6) becomes

[TABLE]

On the other hand, by direct differentiation of $\lambda(y)$ in (11), we can get $\lambda^{\prime}(y)=\frac{q\phi(y)}{\phi(\lambda(y))}$, where $\phi$ is the density function of standard Gaussian. It is not hard to verify $\lambda^{\prime}(y)\in(0,1]$ when $y\geq 0$. Therefore, $y\mapsto y-\lambda(y)$ is non-decreasing and 1-Lipschitz on $\mathbb{R}_{\geq 0}$. On the other hand, $\lambda(0)=\Phi^{-1}(1-\frac{q}{2})\geq 0$. Then following the same argument in Example 1, we get \eta(y;\mu_{Y},\mu_{\Lambda})=\text{sign}(y)\max\big{(}|y|-\lambda(|y|),0\big{)}.

Remark 1

More generally, we can show when $Y$ has a density supported on $\mathbb{R}$ and $y\mapsto y-F_{\Lambda}^{-1}(F_{|Y|}(y))$ is non-decreasing and 1-Lipschitz on $\mathbb{R}_{\geq 0}$, then \eta(y;\mu_{Y},\mu_{\Lambda})=\text{sign}(y)\max\big{(}|y|-F_{\Lambda}^{-1}\big{(}F_{|Y|}(|y|)\big{)},0\big{)}. In some sense, F_{\Lambda}^{-1}\big{(}F_{|Y|}(|y|)\big{)} can be viewed as the equivalent regularization function. This equivalent regularization is adaptive to $y$. As a comparison, the regularization is a constant $\lambda$ in the LASSO case.

III Asymptotic Characterization of SLOPE

Based on the asymptotic separability properties established in the last section, we are now ready to tackle the original optimization problem (2). We are going to obtain the precise characterizations of SLOPE in both estimation and variable selection problems.

III-A Technical Assumptions

Our results are proved under the following assumptions:

1. (A.1)

The number of observations grows in proportion to $p$: $n^{(p)}/p\to\delta\in(0,\infty)$. 2. (A.2)

The number of nonzero elements in $\boldsymbol{\beta}^{(p)}$ grows in proportion to $p$: $r_{0}^{(p)}/p\to\rho\in[0,1]$. 3. (A.3)

The elements of $\boldsymbol{A}^{(p)}$ are i.i.d. Gaussian distribution: $A_{ij}^{(p)}\overset{i.i.d.}{\sim}\mathcal{N}(0,\,\frac{1}{n})$. 4. (A.4)

$\{\boldsymbol{\beta}^{(p)}\}_{p\in\mathbb{N}}$, $\{\boldsymbol{w}^{(p)}\}_{p\in\mathbb{N}}$ and $\{\boldsymbol{\lambda}^{(p)}\}_{p\in\mathbb{N}}$ are converging sequences. The limiting measures are denoted by $\mu_{B}$, $\mu_{W}$ and $\mu_{\Lambda}$, respectively. In addition, $\mathbb{P}(B\neq 0)=\rho$, $\sigma_{w}^{2}=\mathbb{E}[W^{2}]>0$ and $\mathbb{P}(\Lambda\neq 0)>0$ when $\delta\leq 1$, where the probability $\mathbb{P}(\cdot)$ and the expectations $\mathbb{E}[\cdot]$ are all computed with respect to the limiting measures.

III-B Asymptotic Performance of Estimation

The main goal of this section is to derive the limiting MSE of SLOPE: $\lim_{p\to\infty}\frac{1}{p}\|\widehat{\boldsymbol{\beta}}-\boldsymbol{\beta}\|^{2}$. As in [10], we are going to prove a more general result, which characterizes the joint empirical measure of $(\widehat{\boldsymbol{\beta}},\,\boldsymbol{\beta})$ through its action on pseudo-Lipschiz functions.

Definition 1** (Pseudo-Lipschiz function)**

A function $\psi:\,\mathbb{R}^{2}\to\mathbb{R}$ is called pseudo-Lipschiz if $|\psi(\boldsymbol{x})-\psi(\boldsymbol{y})|\leq L(1+\|\boldsymbol{x}\|+\|\boldsymbol{y}\|)\|\boldsymbol{x}-\boldsymbol{y}\|$ for all $\boldsymbol{x},\,\boldsymbol{y}\in\mathbb{R}^{2}$, where $L$ is a positive constant.

To compute the limiting MSE, we just need to let $\psi(\boldsymbol{x})=(x_{1}-x_{2})^{2}$, which is a pseudo-Lipschiz function by the above definition. The general theorem is as follows, whose proof is deferred to Sec. V-B.

Theorem 1

Assume (A.1) – (A.4) hold. For any pseudo-Lipschiz function $\psi$, we have

[TABLE]

where $Y_{*}=B+\sigma_{*}H$ with $B\sim\mu_{B}$, $H\sim\mathcal{N}(0,1)$ independent and $\eta$ is the limiting scalar function defined in Proposition 1. In the above, the scalar pair $(\sigma_{*},\,\tau_{*})$ is the unique solution of the following equations:

[TABLE]

Theorem 1 essentially says that the joint empirical measure of $(\widehat{\boldsymbol{\beta}}^{(p)},\,\boldsymbol{\beta}^{(p)})$ converges to the law of $(\eta(Y_{*};\,\mu_{Y_{*}},\mu_{\tau_{*}\Lambda}),B)$. This means that although the original problem (2) is high-dimensional, its asymptotic performance can be succinctly captured by merely two scalars random variables. From (12) and (13), we know the limiting MSE equals to

[TABLE]

Readers familiar with the asymptotic analysis of LASSO will recognize that the forms of (13) and (14) look identical to the results of LASSO obtained in [10, 50]. Indeed, the proof of Theorem 1 directly applies the framework of analyzing LASSO asymptotics using convex Gaussian min-max theorem (CMGT) [26, 29, 50]. In a nutshell, the CGMT framework builds a connection between the asymptotics of the original high-dimensional problem (2) and the optimal solution of the following two-dimensional minimax problem:

[TABLE]

where $\boldsymbol{\beta}$ is the true signal vector in (1), $\boldsymbol{h}\sim\mathcal{N}(\boldsymbol{0},\boldsymbol{I}_{p})$ and $\mathcal{M}_{\boldsymbol{\lambda}}(\cdot;\cdot)$ is the Moreau envelope defined in (5). In fact, equation (13) and (14) corresponds to the first-order optimality condition of (16). Proposition 1 enables us to justify and explicitly compute the limit in (16), as well as the first-order derivatives $\frac{\partial\mathcal{F}(\sigma,\theta)}{\partial\sigma}$ and $\frac{\partial\mathcal{F}(\sigma,\theta)}{\partial\theta}$, which are crucial in obtaining the optimal point of (16).

III-C Asymptotic Performance of Variable Selection

Next we study the asymptotic performance of SLOPE, when it is used as a variable selection methodology. Under this setting, the goal is to accurately select all the non-zero coordinates of $\boldsymbol{\beta}$. Based on SLOPE estimate, we select the non-zero coordinates of estimate $\widehat{\boldsymbol{\beta}}$. Ideally, we hope that the selected set includes the non-zero coordinates of $\boldsymbol{\beta}$, while do not contain zero coordinates of $\boldsymbol{\beta}$. The usual performance metrics for this task include Type-I error, power, false discovery rate (FDR), etc. Most of these performance metrics can be expressed as a function of the spasiry level $r_{0}^{(p)}$ and the following quantities

[TABLE]

where $R_{0}^{(p)}$ and $V^{(p)}$ are the proportions of discoveries and false discoveries. In the following, we will adopt Type-I error and power as our performance metrics, which can be written as

[TABLE]

In order to study the asymptotics of these testing statistics, we need to obtain the limits of $R_{0}^{(p)}$ and $V^{(p)}$ in (17).

Note that the test functions involved in (17) ($\mathbb{I}_{x=0}$ and $\mathbb{I}_{x\neq 0,y=0}$) are discontinuous, so we can not directly apply (12) in Theorem 1 to compute $\lim_{p\to\infty}R_{0}^{(p)}$ and $\lim_{p\to\infty}V^{(p)}$. Further justifications are needed to obtain companion results for the testing-related statistics in (17). Before delving into techincal descriptions, we first show that counter examples do exist where the quantities in (17) fail to converge, while the assumptions in Theorem 1 are still satisfied. This is different from the LASSO case, where the prediction (12) is shown to be still correct for the above non-smooth indicator functions [9].

Example 3** (A counter example)**

Consider $\mu_{B}$ being a spike-and-slab distribution: $\mu_{B}=0.5\cdot\delta_{0}+0.5\cdot\mathcal{N}(1,0.5^{2})$ and $\{\beta_{i}\}_{i\in[p]}$ are i.i.d. generated from $\mu_{B}$. Let $(\sigma_{*},\tau_{*})$ be the solution of (13)-(14) in the LASSO case, where $\mathbb{P}(\Lambda=1)=1$. Then we construct the following class of distribution of $\Lambda$, parameterized by $\vartheta\in[0,1]$:

[TABLE]

where $Y_{*}=B+\sigma_{*}H$. Here $\vartheta$ is a tuning parameter and $\vartheta=1$ corresponding to the LASSO regularization. In Fig. 3, we plot the empirical $R_{0}^{(p)}$ and MSEs under different values of $\vartheta$. It can be seen from Fig. 3b that for different values of $\vartheta$, the empirical MSEs all concentrate around the predicted values from Theorem 1, when $\Lambda=1$. On the contrary, from Fig. 3a we can find when $\vartheta<1$, $R_{0}^{(p)}$ does not converge to \mathbb{P}\big{(}\eta(Y_{*})=0\big{)}, which is the limit indicated by Theorem 1. Moreover, as $\vartheta$ becomes smaller, the SLOPE estimator becomes less conservative and the variances of $R_{0}^{(p)}$ become increasingly notable. Also we can see $R_{0}^{(p)}$ does converge to \mathbb{P}\big{(}\eta(Y_{*})=0\big{)}, when $\vartheta=1$.

We will explain the logic behind the construction of $\Lambda_{\vartheta}$ in Remark 2 below. The counter example above suggests that some additional constraints are needed, so that the testing statistics in (17) have well-defined limits and (12) can be used to compute these limits. It turns out that we just need one more condition to guarantee their convergence.

Proposition 2

Under the same settings as Theorem 1, define $q_{0}^{*}\overset{\text{def}}{=}\mathbb{P}(\eta(Y_{*})=0).$ If the following condition holds:

1. (R.1)

$q_{0}^{*}=0$* or for any $t\in[0,q_{0}^{*})$, $\int_{t}^{q_{0}^{*}}F_{|Y_{*}|}^{-1}(u)du<\int_{t}^{q_{0}^{*}}F_{\tau_{*}\Lambda}^{-1}(u)du$,*

*then we have *

[TABLE]

where $R_{0}^{(p)}$ and $V^{(p)}$ are defined in (17).

The proof of Proposition 2 will be provided in Appendix V-C, along with some explanations for condition (R.1) (see Remark 10).

Remark 2

In fact, $\Lambda_{\vartheta}$ in (19) is constructed so that condition (R.1) is violated for all $\vartheta<1$. One can easily check that under the setting of Example 3, we have $q_{0}^{*}=F_{|Y_{*}|}(\tau_{*})>0$. From (19) we can get $F_{\tau_{*}\Lambda_{\vartheta}}^{-1}(u)=F_{|Y_{*}|}^{-1}(u)$, for all $u\in[F_{|Y_{*}|}(\vartheta\tau_{*}),F_{|Y_{*}|}(\tau_{*})]$. Also due to the fact that $Y_{*}$ is supported on $\mathbb{R}$, we have $F_{|Y_{*}|}(\vartheta\tau_{*})<F_{|Y_{*}|}(\tau_{*})$, when $\vartheta<1$. Therefore, $\int_{t}^{q_{0}^{*}}F_{|Y_{*}|}^{-1}(u)du=\int_{t}^{q_{0}^{*}}F_{\tau_{*}\Lambda_{\vartheta}}^{-1}(u)du$ for any $t\in[F_{|Y_{*}|}(\vartheta\tau_{*}),q_{0}^{*})$, where we have used $q_{0}^{*}=F_{|Y_{*}|}(\tau_{*})$. This violates condition (R.1). On the other hand, we can also check when $\vartheta=1$, i.e., in the LASSO case, condition (R.1) is satisfied. Indeed, in this case $\Lambda_{\vartheta}=1$ and $F_{\tau_{*}\Lambda_{\vartheta}}^{-1}(u)=F_{\tau_{*}}^{-1}(u)=\tau_{*}$ for any $u\in[0,q_{0}^{*}]$. Besides, since $q_{0}^{*}=F_{|Y_{*}|}(\tau_{*})>0$ and $Y_{*}$ is supported on $\mathbb{R}$, we get $F_{|Y_{*}|}^{-1}(u)<\tau_{*}$ for any $u\in[0,q_{0}^{*})$. Therefore, $\int_{t}^{q_{0}^{*}}F_{|Y_{*}|}^{-1}(u)du<\int_{t}^{q_{0}^{*}}F_{\tau_{*}\Lambda_{\vartheta}}^{-1}(u)du$ for any $t\in[0,q_{0}^{*})$.

Remark 3

*In Example 3, a superficial reason for R_{0}^{(p)}\not\to\mathbb{P}\big{(}\eta(Y_{*})=0\big{)} when $\vartheta<1$ is that $\boldsymbol{\lambda}$ generated from such $\Lambda$ will lead to many pseudo-zero entries in $\widehat{\boldsymbol{\beta}}$, i.e., entries that are very closed to 0, but not strictly 0. This is illustrated in Fig. 3c and 3d. In practice, the pseudo-zero effects can be mitigated by employing post-screening to $\widehat{\boldsymbol{\beta}}$. This is done by first specifying a threshold $h>0$ and then setting all the entries in $\widehat{\boldsymbol{\beta}}$ with $|\hat{\beta}_{i}|<h$ to be zero. However, this creates a new problem of choosing the appropriate $h$. Our claim is that this problem can be completely avoided by adding an extra constraint on the regularizing sequence. Moreover, as will be clarified in Sec. IV-B, this additional constraint will not harm the diversity of our design choices. *

Based on Proposition 2, we can now compute the limiting Type-I error and power of SLOPE.

Corollary 1

When $\mathbb{P}(B=0)\in(0,1)$, we have

[TABLE]

and

[TABLE]

where $y_{\text{th}}^{*}=\sup_{y\geq 0}\{y\mid\eta(y;\mu_{Y_{*}},\mu_{\tau_{*}\Lambda})=0\}$.

The proof of Corollary 1 directly follows from (18), (20) and the assumption that $r_{0}^{(p)}/p\to\mathbb{P}(B\neq 0)$. Formulas (21) and (22) will be useful in Sec. IV-B, where we analyze the optimal performance of SLOPE for variable selection.

Remark 4

In Corollary 1, we require that $\mathbb{P}(B=0)\in(0,1)$. This means asymptotically, the proportions of zero and non-zero entries of $\boldsymbol{\beta}$ are both non-vanishing. We need this assumption on the distribution of $B$, because otherwise the limiting formula of Type-I error and power will involve with $\frac{0}{0}$ term, when we apply (18). This is beyond the scope of asymptotic setting considered in this paper.

IV Fundamental Limits and Optimal Regularization

Armed with the asymptotic characterizations in Theorem 1 and Proposition 2, we are now ready to analyze the optimal performance of SLOPE in both estimation and variable selection setting.

IV-A Estimation with Minimum MSE

We first turn to the problem of finding the minimum MSE achievable by SLOPE estimator and the corresponding optimal regularization. In the current asymptotic setting, this can be formulated as follows:

[TABLE]

where $\mathcal{P}_{\Lambda}\overset{\text{def}}{=}\{\mu_{\Lambda}\mid\mu_{\Lambda}\in\mathcal{P}_{2}(\mathbb{R})\text{ and }\mathbb{P}(\Lambda\neq 0)>0,\text{ when }\delta\leq 1\}$ is the admissible set of $\mu_{\Lambda}$, under which the asymptotic characterization in Theorem 1 is valid. By (15), solving (23) is equivalent to solving

[TABLE]

In the current context, $\sigma_{*}$ should be understood as a function of $\mu_{\Lambda}$, but for notational simplicity, we will drop its dependency on $\mu_{\Lambda}$, when doing so causes no confusion.

Note that $\sigma_{*}$ is determined by $\mu_{\Lambda}$ implicitly through the nonlinear fixed point equation (13)-(14), so a direct optimization over $\mu_{\Lambda}$ as in (24) is not viable. To proceed, a key observation from (13)-(14) is that the influence of $\mu_{\Lambda}$ is exerted only through the limiting scalar function $\eta$. In light of this, (24) can be alternatively solved via the following two-step scheme:

1. Step 1.

Search over all realizable $\eta$ such that there exists $\sigma,\tau>0$ satisfying

[TABLE]

and find optimal $\eta^{\star}$ that yields the minimum feasible $\sigma$. Denote the corresponding solution of (25)-(26) as $(\sigma^{\star},\tau^{\star})$. 2. Step 2.

Find corresponding $\mu_{\Lambda}$ such that $\eta(y;\mu_{B+\sigma^{\star}H},\mu_{\tau^{\star}\Lambda})=\eta^{\star}(y)$.

Note that in Step 1, $\eta$ is treated as an optimization variable that do not depend on other parameters, which greatly simplifies the original formulation (24). However, to implement this scheme, we still need to guarantee two things. First, the realizable set of $\eta$ (as required in Step 1) needs to be decided. Second, for any realizable $\eta$, the corresponding $\Lambda$ can be efficiently computed. These are both addressed in the following result.

Proposition 3

For a probability measure $\mu_{Y}\in\mathcal{P}_{2}(\mathbb{R})$, define

[TABLE]

where $\eta(\cdot\,;\mu_{Y},\mu_{\Lambda})$ is the limiting scalar function in Proposition 1. Then for any $\mu_{Y}\in\mathcal{P}_{2}(\mathbb{R})$, we have $\mathcal{M}_{\mu_{Y}}=\mathcal{I}$. Correspondingly, for any $f(y)\in\mathcal{I}$, we can take $\Lambda\sim|Y|-f(|Y|)$, with $Y\sim\mu_{Y}$, so that $\eta(y\,;\mu_{Y},\mu_{\Lambda})=f(y)$.

The proof of Proposition 3 will be presented in Appendix -L. It is the key ingredient in proving our optimality results. It shows that, with different choices of $\mu_{\Lambda}$, one can reach any non-decreasing and odd function that is Lipschitz continuous with constant 1. Clearly, the soft-thresholding functions associated with LASSO belongs to $\mathcal{M}_{\mu_{Y}}$, but the set $\mathcal{M}_{\mu_{Y}}$ is much richer. This is how SLOPE generalizes LASSO: it allows for more degrees of freedom in the regularization.

Based on Proposition 3, we are now ready to show the two-step scheme sketched above indeed yield a computationally feasible procedure to obtain the minimum MSE and the optimal $\mu_{\Lambda}$. Before that, we first introduce the following function:

[TABLE]

We will see for any $\sigma>0$, problem (28) is convex and there exists a unique optimal solution. Given $\mathcal{L}(\sigma)$, we then introduce the following equation on $\sigma$:

[TABLE]

As is shown in Proposition 4 below, the minimum limiting MSE is closely related with the minimum solution of equation (29).

Proposition 4

Under the same setting as Theorem 1, we have

• (a)

For any $\sigma>0$, problem (28) is convex and there exists a unique optimal solution $f_{\sigma}\in\mathcal{I}$.

• (b)

$\mathcal{L}(\sigma)$* defined in (28) is continuous on $\mathbb{R}_{>0}$ and equation (29) always has a solution. The minimum solution \sigma_{0}\in\big{[}\sigma_{w},\,\sqrt{\sigma_{w}^{2}+\delta^{-1}\mathbb{E}B^{2}}\big{]}.*

• (c)

Define $Y_{0}:=B+\sigma_{0}H$ and $\tau_{0}:=\left[1-{\delta^{-1}}{\mathbb{E}f_{\sigma_{0}}^{\prime}(Y_{0})}\right]^{-1}.$ It always holds that

[TABLE]

Moreover, if \delta^{-1}\mathbb{E}\big{[}f_{\sigma_{0}}^{\prime}(Y_{0})\big{]}<1, the equality in (30) can be attained, when $\mu_{\Lambda}$ is the law of

[TABLE]

The proof of Proposition 4 is deferred to Appendix V-D. To solve the infinite-dimensional optimization problem (28) in practice, we can discretize over $\mathbb{R}$ and obtain a finite-dimensional approximation. Naturally, this finite-dimensional problem is still convex. In our simulation, we use an approximation with 2048 grids.

We have a couple of comments regarding Proposition 4 as follows.

Remark 5** (Interpretation of $\mathcal{L}(\sigma)$)**

Consider the optimization in (28):

[TABLE]

and we neglect the constraint $\delta^{-1}\mathbb{E}[f^{\prime}(B+\sigma H)]\leq 1$ for a moment. From Proposition 1 and Proposition 3 we know minimization in (32) is equivalent to

[TABLE]

where $\boldsymbol{h}\sim\mathcal{N}(\boldsymbol{0},\boldsymbol{I}_{p})$ and $\mu_{\Lambda}=\lim_{p\to\infty}\mu_{\boldsymbol{\lambda}}$. In other words, we are estimating $\boldsymbol{\beta}$ from the noisy observation: $\boldsymbol{y}=\boldsymbol{\beta}+\sigma\boldsymbol{h}$ using SLOPE and we want to find the optimal regularization (specified by its limiting distribution $\mu_{\Lambda}$) such that the estimation error of $\boldsymbol{\beta}$ is minimized. Then $\mathcal{L}(\sigma)$ can be understood as the minimum MSE we can achieve, if we put an additional constraint on the average slope of limiting scalar function. On the other hand, if at the optimal solution $f_{\sigma}$, the constraint is inactive, i.e., $\delta^{-1}\mathbb{E}[f_{\sigma}^{\prime}(B+\sigma H)]<1$, then $\mathcal{L}(\sigma)=\inf_{f\in\mathcal{I}}\mathbb{E}[f(B+\sigma H)-B]^{2}$. This can be easily verified as follows. Assume there exists $f_{\star}\in\mathcal{I}$ such that $\mathbb{E}[f_{\star}(B+\sigma H)-B]^{2}<\mathcal{L}(\sigma)$. Then consider the convex combination $f_{t}:=tf_{\star}+(1-t)f_{\sigma}$, for $t\in(0,1)$. Clearly, $f_{t}\in\mathcal{I}$ and it is not hard to check for small enough $t$, $\delta^{-1}\mathbb{E}[f_{t}^{\prime}(B+\sigma H)]\leq 1$. However, due to the convexity of objective function in (28),

[TABLE]

which leads to a contradiction.

Remark 6** (Tightness of lower bound (30))**

We require $\delta^{-1}\mathbb{E}f_{\sigma_{0}}^{\prime}(B+\sigma_{0}H)<1$ so that the lower bound (30) is tight. The question is whether it is possible that $\delta^{-1}\mathbb{E}f_{\sigma_{0}}^{\prime}(B+\sigma_{0}H)=1$. This will not happen when $\delta>1$, since $f_{\sigma_{0}}^{\prime}\leq 1$. When $\delta\leq 1$, we do not have a rigorous proof yet. Numerically, this never happens either. Here we provide an intuitive argument. Suppose for certain configurations of $(\delta,\rho,\sigma_{w},\mu_{B})$, we do have $\delta^{-1}\mathbb{E}f_{\sigma_{0}}^{\prime}(B+\sigma_{0}H)=1$. Under this scenario, let us consider the following approximation of (28) and (29), parameterized by $\varepsilon>0$:

[TABLE]

and

[TABLE]

Denote $\sigma_{0,\varepsilon}$ as the minimum solution of equation (34) and $f_{\varepsilon}$ as the optimal solution of (33), when $\sigma=\sigma_{0,\varepsilon}$. If we take $\mu_{\Lambda}$ to be the law of

[TABLE]

*where $Y_{0,\varepsilon}=B+\sigma_{0,\varepsilon}H$ and \tau_{0,\varepsilon}=\big{[}1-{\delta^{-1}}{\mathbb{E}f_{\varepsilon}^{\prime}(Y_{0,\varepsilon})}\big{]}^{-1}<\infty. Then similar as Proposition 4, it is not hard to show $\lim_{p\to\infty}\frac{1}{p}\|\widehat{\boldsymbol{\beta}}_{\varepsilon}-\boldsymbol{\beta}\|_{2}^{2}=\delta(\sigma_{0,\varepsilon}^{2}-\sigma_{w}^{2})$, where $\widehat{\boldsymbol{\beta}}_{\varepsilon}$ denotes the corresponding estimator. Intuitively, we could also expect $\sigma_{0,\varepsilon}\to\sigma_{0}$ and $\tau_{0,\varepsilon}\to\infty$, as $\varepsilon\to 0$. This implies the MSE can be made arbitrarily close to the lower bound (30) using a sequence $\{\mu_{\Lambda,\varepsilon}\}_{\varepsilon>0}$ which converges to the probability mass at 0 as $\varepsilon\to 0$. Recall that we have assumed $\sigma_{w}>0$, so this means the optimal regularization in a noisy overparameterized linear model should be vanishingly small, which is not likely the case. *

Remark 7** (Comparison with [46, 49])**

In [46, 49], the authors also analyze the problem of optimal estimation in the linear model (1) with i.i.d. Gaussian design. For the convenience of comparison, here we rephrase their results in our notations. The optimality they consider is with respect to the following class of estimator:

[TABLE]

where

[TABLE]

The optimal estimation within the class of $\mathcal{C}_{p}$ is formulated as:

[TABLE]

where $\mathcal{W}_{p}$ is some set that ensures $\widehat{\boldsymbol{\beta}}$ is unique 222In fact, $\mathcal{W}_{p}$ corresponds to the tightness condition \delta^{-1}\mathbb{E}\big{[}f_{\sigma_{0}}^{\prime}(B+\sigma_{0}H)\big{]}<1 in Proposition 4 (c).. One of their main results states that under certain conditions, the minimum achievable limiting MSE defined in (37) satisfies: $\text{MSE}_{\text{ cvx}}\geq\delta(\sigma_{\text{cvx}}^{2}-\sigma_{w}^{2})$, where

[TABLE]

*with $\mathcal{J}\overset{\text{def}}{=}\{f:\mathbb{R}\to\mathbb{R}\mid f\text{ is non-decreasing and 1-Lipschitz continuous}\}$. Comparing their results with ours, we can find the lower bounds in both settings follow the same type of characterization. Specifically, lying at the heart of this characterization is an optimization problem: $\inf_{f\in\mathcal{F}}\mathbb{E}[f(B+\sigma H)-B]^{2}$, which aims at finding the optimal estimator $f$ of $B$ under the noisy observation $B+\sigma H$. The only difference is on the feasible set $\mathcal{F}$: $\mathcal{F}=\mathcal{J}$ in (38), while $\mathcal{F}=\mathcal{I}\subset\mathcal{J}$ in (28). This agreement is not a coincidence, but related with the fact that the proximal operator of all functions in $\mathcal{C}_{p}$ is asymptotically separable as proved in [49]. In fact, $f$ corresponds to the limiting proximal operator of the regularizer $r_{p}$. In our settings, $r_{p}$ is chosen from the set of all possible sorted $\ell_{1}$ norms (denoted by $\mathcal{S}_{p}$), while in their settings, it is chosen from the set $\mathcal{C}_{p}$. Correspondingly, $\mathcal{I}$ is the set of all limiting proximal operators associated with $\mathcal{S}_{p}$ and $\mathcal{J}$ is the one associated with $\mathcal{C}_{p}$. It is not hard to check $\mathcal{S}_{p}\subset\mathcal{C}_{p}$ and consequently, we have $\mathcal{I}\subset\mathcal{J}$. *

In Fig. 4, we compare the MSEs achieved by different regularizing sequences (LASSO, BHq and oracle optimal design), at different SNR and sparsity levels. Since we are concerned with oracle optimality, for fair comparison, we search through the parameters of the BHq and LASSO sequences (in particular, $q$ for BHq and $\lambda$ for LASSO) and report the minimum MSEs that can be achieved. The solid curves correspond to the theoretical MSEs predicted by Theorem 1 and Proposition 4. Note that the empirical MSEs match well with theoretical predictions 333Here, the MSEs of LASSO and BHq are obtained by optimizing over the parameters $\lambda$ and $q$, so strictly speaking, the theoretical curves are valid only if a stronger uniform convergence result holds. The uniform convergence for LASSO case is proved in [51, 50] and we conjecture that it also holds true for BHq sequences.. It is also observed that under each setting, the MSEs of different regularizing sequences are all above the lower bound obtained in (30) (red curve in the figure). Also we can see this lower bound can be attained when the limiting empirical distribution of $\boldsymbol{\lambda}$ follows prescribed optimal distribution (31). We also have the following findings:

As can be seen from Fig. 4a and Fig. 4b, when $\rho$ is small, LASSO performs well and the corresponding MSEs almost match the theoretical lower bound, across different values of SNR. However, its performance degrades faster than the other two sequences, as $\rho$ grows. This is because LASSO’s penalization is not adaptive to the underlying sparsity levels and it incurs higher bias under larger $\rho$ [7]. 2. 2.

From Fig. 4b and Fig. 4c, we can find that at low SNR regimes, the BHq sequence can lead to comparable performance as the optimal design. However, at higher SNR regimes, the optimal design notably outperforms the BHq sequence. To explain this phenomenon, we plot in Fig. 5 the empirical distributions of the $\boldsymbol{\lambda}$-sequences associated with the optimal design and the BHq design, respectively. It turns out that, in the low SNR case, the optimal design and BHq have similar distributions, while at higher SNRs, the distribution of the optimal design is close to a mixture of a delta mass and uniform distribution.

IV-B Variable Selection with Maximum Power

Next we consider using SLOPE for variable selection. Our goal is to find the optimal regularizing sequence to achieve the highest possible power, under a given level of type-I error $\alpha$, formulated as:

[TABLE]

where {\widetilde{\mathcal{P}}_{\Lambda}}\overset{\text{def}}{=}\{\Lambda\in\mathcal{P}_{\Lambda}:\text{\ref{enu:R_strict_majorization} is satisfied}\} is the admissible set of $\mu_{\Lambda}$, with which the limits in (39) exist. In light of (21) and (22), if $\mathbb{P}(B=0)\in(0,1)$, optimization problem (39) is equivalent to:

[TABLE]

where $y_{\text{th}}^{*}=\sup_{y\geq 0}\{y\mid\eta(y;\mu_{Y_{*}},\mu_{\tau_{*}\Lambda})=0\}$. Comparing the admissible set $\widetilde{\mathcal{P}}_{\Lambda}$ with $\mathcal{P}_{\Lambda}$ in (24), it can be seen the only difference is that here we need an additional condition (R.1) to ensure the limits of Type-I error and Power both exist (see Proposition 2).

To state our results, we first introduce the following function, which is the counterpart of (28).

[TABLE]

where $\alpha\in[0,1]$ is a prescribed Type-I error level and \mathcal{F}_{\alpha,\sigma}\overset{\text{def}}{=}\big{\{}f(y):f(y)=0\text{ for }|y|\leq\Phi^{-1}(1-\tfrac{\alpha}{2})\sigma\big{\}}. Similar as Proposition 4, we will see that the maximum power achievable by SLOPE under Type-I error level $\alpha$ is related with the following equation:

[TABLE]

where $\mathcal{L}_{\alpha}(\sigma)$ is the function defined in (41).

We are now ready to state our main optimality results for variable selection.

Proposition 5

Under the same setting as Proposition 2, assume $\mathbb{P}(B=0)\in(0,1)$. Then we have

• (a)

For any $\alpha\in[0,1]$ and $\sigma>0$, problem (41) is convex and there exists a unique optimal solution $f_{\alpha,\sigma}\in\mathcal{I}$.

• (b)

For any $\alpha\in[0,1]$, $\mathcal{L}_{\alpha}(\sigma)$ is continuous on $\mathbb{R}_{>0}$ and equation (42) always has a solution. The minimum solution \sigma_{0,\alpha}\in\big{[}\sigma_{w},\,\sqrt{\sigma_{w}^{2}+\delta^{-1}\mathbb{E}B^{2}}\big{]}.

• (c)

Let $Y_{0,\alpha}:=B+\sigma_{0,\alpha}H$ and $f_{\alpha}:=f_{\alpha,\sigma_{0,\alpha}}$. If $\lim_{p\to\infty}\text{Type-I error}\leq\alpha$, then

[TABLE]

Moreover, if \delta^{-1}\mathbb{E}\big{[}f_{\alpha}^{\prime}(Y_{0,\alpha})\big{]}<1 and $y_{0,\alpha}=\Phi^{-1}(1-\frac{\alpha}{2})\sigma_{0,\alpha}$, the upper bound in (43) can be attained by $\mu_{\Lambda}=\mu_{\text{opt},\alpha}$, with $\mu_{\text{opt},\alpha}$ being the law of

[TABLE]

Here, $y_{0,\alpha}=\sup_{y\geq 0}\{y\mid f_{\alpha}(y)=0\}$ and \tau_{0,\alpha}=\big{[}1-\delta^{-1}\mathbb{E}f_{\alpha}^{\prime}(Y_{0,\alpha})\big{]}^{-1}.

The proof of Proposition 5, which is similar to that of Proposition 4, will be given in Sec. V-E. A key step is to show the realizable set of $\eta$ in the variable selection setting is still equal to $\mathcal{I}$ (see Lemma 20 in Appendix -N), although the admissible set of $\mu_{\Lambda}$ is replaced by $\widetilde{\mathcal{P}}_{\Lambda}$, which is a subset of $\mathcal{P}_{\Lambda}$ in the estimation setting.

Remark 8

Comparing the results in Proposition 4 and Proposition 5, we can find that although at the beginning, we are dealing with two different problems (the objective of the first one is minimizing the MSE, while the other is on maximizing the power under a given Type-I error), we end up with two procedures of very similar natures. Both problems can finally be converted into a formulation involving finding the optimal estimation of $\boldsymbol{\beta}$ that can be achieved by SLOPE under the observation $\boldsymbol{y}=\boldsymbol{\beta}+\sigma\boldsymbol{h}$, with $\boldsymbol{h}\sim\mathcal{N}(\boldsymbol{0},\boldsymbol{I})$. The only difference is that in the second problem, we need to enforce an additional restriction on the regularization sequence $\boldsymbol{\lambda}$ to ensure the Type-I error is below certain threshold $\alpha$.

Remark 9** (Tightness of upper bound (43))**

The tightness of the upper bound for power relies on the conditions: \delta^{-1}\mathbb{E}\big{[}f_{\alpha}^{\prime}(Y_{0,\alpha})\big{]}<1 and $y_{0,\alpha}=\Phi^{-1}(1-\frac{\alpha}{2})\sigma_{0,\alpha}$. Numerically they hold under all the settings considered. We conjecture that within our assumptions, this condition always hold and the upper bound (43) is tight.

In Fig.6, we compare the variable selection performance achieved by the optimal regularization with that of LASSO and BHq sequences. We show both theoretical ROC curves and the empirical power under given Type-I error levels. Here each empirical $(\text{Type-I error},\text{power})$ pair is generated by first fixing all the parameters (including the tuning parameters such as $\lambda$ and $q$) and then averaging over 20 independent trials. It can seen that the empirical results match well with the theoretical predictions (solid curves in the figures) and the optimal design of regularization dominates the other two regularizing sequences. We also have the following observations:

In all cases, the theoretical upper bounds on power (43) can be achieved by choosing $\mu_{\Lambda}$ to be the law of (44). 2. 2.

The performance of LASSO is closed to the fundamental limit at low sparsity and high SNR regimes, while its performance is significantly degraded as sparsity grows higher or SNR grows lower. In particular, we can find in such cases, the maximum power achievable by LASSO is less than 1. This phenomenon is also discussed in [2, 7, 11] and it is inherently connected with the so-called “noise-sensitivity ” phase transition [52]. In comparison, the optimal and BHq sequences can both reach power 1, after Type-I errors are above certain thresholds. 3. 3.

Complementary to LASSO, the performance of BHq sequences is closed to the theoretical upper bounds at low SNRs or large sparsity levels, while it deviates from the upper bounds in other scenarios.

V Proof of Main Results

V-A Asymptotic Separability

In this section, we are going to prove Proposition 1.

From (5) we have the following scaling property: $\mathcal{M}_{\boldsymbol{\lambda}}(\boldsymbol{y};\tau)=\frac{1}{\tau}\mathcal{M}_{\tau\boldsymbol{\lambda}}(\boldsymbol{y};1)$. On the other hand, for any $\tau>0$, if $\boldsymbol{\lambda}$ is a converging sequence with limiting measure $\mu_{\Lambda}$, it is not hard to show $\tau\boldsymbol{\lambda}$ is also a converging sequence, with limiting measure $\mu_{\tau\Lambda}$. Thus, to study the asymptotic limit of (5) under $(\boldsymbol{y},\boldsymbol{\lambda},\tau)$, it suffices to consider $(\boldsymbol{y},\tau\boldsymbol{\lambda},1)$. As a result, without loss of generality, we will assume $\tau=1$ in the rest of our proof.

V-A1 Some preliminary facts about SLOPE

The asymptotic separability stems from the following unique properties of the SLOPE proximal minimization problem (5), which are proved in [9, Sec. 2].

Fact 1

*For any $\boldsymbol{\lambda},\boldsymbol{y}\in\mathbb{R}^{p}$, with $\lambda_{i}\geq 0$, for all $i\in[p]$, it holds that ***

1. (i)

(Sign consistency)* For any $i\in[p]$, $[\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{y})]_{i}$ has the same sign as $y_{i}$. Moreover,*

[TABLE] 2. (ii)

(Permutation-invariance)* For any permutation matrix $\Pi$, $\Pi\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{y})=\text{Prox}_{\boldsymbol{\lambda}}(\Pi\boldsymbol{y}).$* 3. (iii)

(Monotonicity and Lipschitz continuity)* For any $i,j\in[p]$, if $y_{i}\leq y_{j}$, then $0\leq[\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{y})]_{j}-[\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{y})]_{i}\leq y_{j}-y_{i}$ and for any $y_{i}$, $[\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{y})]_{i}\leq|y_{i}|$.*

An immediate yet important implication of Fact 1 is the following lemma:

Lemma 1

For any $\boldsymbol{\lambda},\boldsymbol{y}\in\mathbb{R}^{p}$, with $\lambda_{i}\geq 0$, there always exists an odd, non-decreasing and 1-Lipschitz function $g_{p}$ such that for all $i\in[p]$, $g_{p}(y_{i})=[\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{y})]_{i}$.

The proof of Lemma 1 is given in Appendix -A. By Lemma 1 we know $\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{y})$ is actually the restriction of a function $g_{p}\in\mathcal{I}$ onto the support of $\mu_{\boldsymbol{y}}$. Moreover, from the permutation invariance property (Fact 1 (ii)), such $g_{p}$ is only determined by the empirical measure $\mu_{\boldsymbol{\lambda}}$ and $\mu_{\boldsymbol{y}}$. We could expect if $\mu_{\boldsymbol{\lambda}}$ and $\mu_{\boldsymbol{y}}$ both converge to some limiting distributions, $g_{p}$ will also converge to certain limiting scalar function. This is exactly the essential meaning of asymptotic separability.

Before proceeding, let us take a look at a numerical justification shown in Fig. 7. Here we choose $g_{p}$ to be the linear interpolation of the following set of points: $\{(|y_{i}|,|\hat{y}_{i}|),(-|y_{i}|,-|\hat{y}_{i}|)\}_{i=1}^{p}\cup(0,0),$ where $\hat{\boldsymbol{y}}:=\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{y})$. It is easy to check (as is shown in the proof of Lemma 1) such linear interpolation is a qualified candidate for $g_{p}$ in Lemma 1. We compare it with the limiting scalar function $\eta(y)$ predicted by Proposition 1. It is clear that as $p$ becomes larger, $g_{p}(y)$ gets increasingly close to $\eta(y)$.

V-A2 An equivalent form of (5)

Based on Lemma 1, we then go on to show the equivalence between (5) and the following problem:

[TABLE]

This is formalized in the following lemma, whose proof is given in Appendix -B.

Lemma 2

Denote $\mathcal{M}_{\boldsymbol{\lambda}}^{*}(\boldsymbol{y})$ as the optimal value of (45). Then it holds that $\frac{\mathcal{M}_{\boldsymbol{\lambda}}(\boldsymbol{y};1)}{p}=\mathcal{M}_{\boldsymbol{\lambda}}^{*}(\boldsymbol{y})$. Besides, any optimal solution $g_{p}^{*}(y)$ of (45) satisfies: $g_{p}^{*}(\boldsymbol{y})=\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{y})$.

Comparing (6) and (45), it could be now understood that the optimization in (6) is the limit of (45), as $\mu_{\boldsymbol{\lambda}}\to\mu_{\Lambda}$ and $\mu_{\boldsymbol{y}}\to\mu_{Y}$. Therefore, from Lemma 2, we could expect $\frac{1}{p}\mathcal{M}_{\boldsymbol{\lambda}}(\boldsymbol{y};1)=\mathcal{M}_{\boldsymbol{\lambda}}^{*}(\boldsymbol{y})\to\mathcal{M}_{\mu_{\Lambda}}(\mu_{Y},1)$. On the other hand, $g_{p}^{*}(\cdot)$, which is the optimal solution of (45) should also converge to the optimal solution of (6): $\eta(\cdot;\mu_{Y},\mu_{\Lambda})$. Thus for any $\boldsymbol{\lambda}$, $\boldsymbol{y}$ satisfying $\mu_{\boldsymbol{\lambda}}\approx\mu_{\Lambda}$ and $\mu_{\boldsymbol{y}}\approx\mu_{Y}$, we would have $\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{y})=g_{p}^{*}(\boldsymbol{y})\approx\eta(\boldsymbol{y};\mu_{Y},\mu_{\Lambda})$,* i.e.*, asymptotic separability holds. The final step of the proof is to make the above intuition accurate and rigorous.

V-A3 Taking the limit of (45)

Recall that we have assumed $\tau=1$. For notational simplicity, denote $\mathcal{M}_{\boldsymbol{\lambda}}(\boldsymbol{y}):=\mathcal{M}_{\boldsymbol{\lambda}}(\boldsymbol{y};1)$ and $\mathcal{M}_{\mu_{\Lambda}}(\mu_{Y}):=\mathcal{M}_{\mu_{\Lambda}}(\mu_{Y},1)$. Define $L(g)$ as the objective function of (6), i.e.,

[TABLE]

and $g^{*}$ as the corresponding optimal solution. By Lemma 7 in Appendix -C, we have $\sup_{g\in\mathcal{I}}|L(g)-L_{p}(g)|\to 0$, where $L_{p}(g)$ is the objective function of (45). Therefore, (8) immediately follows, since

[TABLE]

On the other hand,

[TABLE]

where in the last step we use the optimality of $g^{*}$. By the strong convexity of (45), we have

[TABLE]

Combining (47) and (48) gives

[TABLE]

By Lemma 7 again, we have $\mathbb{E}_{\mu_{\boldsymbol{y}}}|g_{p}^{*}(Y)-g^{*}(Y)|^{2}\to 0$, as $p\to\infty$. This is exactly (9), since $g_{p}^{*}(\boldsymbol{y})=\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{y})$ by Lemma 2 and $g^{*}(y)=\eta(y;\mu_{Y},\mu_{\tau\Lambda})$. Finally, the uniqueness of $g^{*}(\cdot)$ (up to a set of measure 0 with respect to $\mu_{Y}$) is proved in Lemma 8. This completes our proof.

V-B Asymptotic Estimation Performance

V-B1 Convex Gaussian Min-max Theorem

Our proof hinges on the Convex Gaussian Min-max Theorem (CGMT). For completeness, we briefly summarize the key idea here. The CGMT studies a minimax optimization problem (PO) of the form:

[TABLE]

where $\mathcal{S}_{\boldsymbol{v}}\subset\mathbb{R}^{p}$, $\mathcal{S}_{\boldsymbol{u}}\subset\mathbb{R}^{n}$ are two compact sets, $\psi:\mathcal{S}_{\boldsymbol{v}}\times\mathcal{S}_{\boldsymbol{u}}\to\mathbb{R}$ is a continuous convex-concave function w.r.t. $(\boldsymbol{v},\boldsymbol{u})$ and $G_{ij}\overset{i.i.d.}{\sim}\mathcal{N}(0,1)$. Problem (49) can be associated with the following auxiliary optimization (AO) problem:

[TABLE]

where $\boldsymbol{g}\sim\mathcal{N}(\mathbf{0},\boldsymbol{I}_{n})$ and $\boldsymbol{h}\sim\mathcal{N}(\mathbf{0},\boldsymbol{I}_{p})$. Roughly speaking, CGMT shows that $\frac{1}{p}\Phi(\boldsymbol{G})\approx\frac{1}{p}\phi(\boldsymbol{g},\boldsymbol{h})$ and the optimal solutions of (49) and (50) have approximately the same empirical distributions in the large $p$ limit. Usually, (AO) is easier to analyze, so it provides a convenient handle for analyzing (PO). For a detailed descriptions, readers can refer to [26, Theorem 3].

V-B2 Proof of Theorem 1

The first step is to recast (2) into the minimax form as in (49). Letting $\boldsymbol{v}=\boldsymbol{x}-\boldsymbol{\beta}$, (2) can be equivalently written as:

[TABLE]

Denote $\widehat{\boldsymbol{v}}\overset{\text{def}}{=}\underset{\boldsymbol{v}}{\text{argmin }}C(\boldsymbol{v})$ and correspondingly, $\widehat{\boldsymbol{\beta}}=\widehat{\boldsymbol{v}}+\boldsymbol{\beta}$. Now (51) has the same form as (49) and the corresponding (AO) is:

[TABLE]

where $\boldsymbol{g}\sim\mathcal{N}(\mathbf{0},\boldsymbol{I}_{n})$ and $\boldsymbol{h}\sim\mathcal{N}(\mathbf{0},\boldsymbol{I}_{p})$. Let $D\subset\mathbb{R}^{p}$ be any closed set. Then by CGMT we can show for any $t\in\mathbb{R}$,

[TABLE]

and if $D$ is also convex,

[TABLE]

The proof of (53) and (54) is the same as [50, Corollary 5.1] and is omitted here. We are going to apply (53) and (54) to prove (12). We will follow the proof steps in [50].

First define the following minimax problem:

[TABLE]

where $\mathcal{F}(\sigma,\theta)\overset{\text{def}}{=}\frac{\theta}{2\sigma}\mathbb{E}[Y-\eta(Y)]^{2}+\int_{0}^{1}F_{\Lambda}^{-1}(u)F_{|\eta(Y)|}^{-1}(u)du,$ with $\eta(\cdot)=\eta(\cdot;\mu_{Y},\mu_{\sigma\Lambda/\theta})$. To prove (12), we adopt the same perturbation argument as in [26, 50, 29]. In particular, for a pseudo-Lipschiz function $\psi(\cdot,\cdot)$, define the following set of $\boldsymbol{v}$:

[TABLE]

where $\nu>0$ and $\mu^{*}$ denotes the joint measure of \big{(}\eta(Y_{*};\mu_{Y_{*}},\mu_{\sigma_{*}\Lambda/\theta_{*}}),B\big{)}. Here $(\sigma_{*},\theta_{*})$ is the optimal solution of (55) and $Y_{*}=B+\sigma_{*}H$, with $H\sim\mathcal{N}(0,1)$ independent of $B\sim\mu_{B}$. Recall that $\widehat{\boldsymbol{\beta}}=\widehat{\boldsymbol{v}}+\boldsymbol{\beta}$, so for any $\nu>0$ and $\varepsilon>0$

[TABLE]

This indicates that if we can show for any $\nu>0$ and some $\varepsilon>0$, $\min_{\boldsymbol{v}\in D_{\nu}}C(\boldsymbol{v})\leq\min_{\boldsymbol{v}}C(\boldsymbol{v})+\varepsilon$ occurs with vanishing probability, then (12) will immediately follow (with $\tau_{*}=\sigma_{*}/\theta_{*}$). In this way, proving (12) is reformulated as the perturbation analysis of $C_{\boldsymbol{\lambda}}(\boldsymbol{v})$, which can be done as follows. For any $\varepsilon\geq 0$ and $K>0$ , we have

[TABLE]

where (a) is due to (53) and (54). Here $\Psi_{*}$ is the optimal value of (55). In Appendix -D, we show all the three probabilities on the RHS of (58) vanish for $K\geq\tfrac{\sigma_{*}}{\sqrt{\delta}}+\tfrac{\theta_{\min}}{4}$ (with $\theta_{\min}$ given in Lemma 14) :

1. (i)

From (88) of Lemma 10, \mathbb{P}\big{(}|\min_{\boldsymbol{v}\in\mathcal{B}_{\sqrt{n}K}}L(\boldsymbol{v})-\Psi_{*}|\geq\varepsilon\big{)}\to 0 for any $\varepsilon>0$. 2. (ii)

From (112) of Lemma 12, \mathbb{P}\big{(}\hat{\boldsymbol{v}}\notin\mathcal{B}_{\sqrt{n}K}\big{)}\to 0. 3. (iii)

From Lemma 11, for any $\nu>0$ there exists $\varepsilon_{0}>0$ such that for any $\varepsilon\leq\varepsilon_{0}$, \mathbb{P}\big{(}\min_{\boldsymbol{v}\in D_{\nu}\bigcap\mathcal{B}_{\sqrt{n}K}}L(\boldsymbol{v})\leq\Psi_{*}+2\varepsilon\big{)}\to 0.

After substituting (i)-(iii) back to (58), we deduce that for any $\nu>0$, there always exist $\varepsilon_{0}>0$ such that for any $\varepsilon\leq\varepsilon_{0}$, the RHS of (57) converges to 0 as $p\to\infty$. Therefore,

[TABLE]

On the other hand, by Lemma 14 in Appendix -J, $(\sigma_{*},\theta_{*})$ is the unique solution of the following fixed point equation of $(\sigma,\theta)$:

[TABLE]

Therefore, letting $\tau_{*}=\sigma_{*}/\theta_{*}$, we can see $(\sigma_{*},\tau_{*})$ is also a solution of (13)-(14). Finally, we show such $(\sigma_{*},\tau_{*})$ is the unique solution of (13)- (14). By Lemma 14, $(\sigma_{*},\theta_{*})$ is the unique solution of (59)-(60) and it satisfies $\sigma_{*}\geq\sigma_{w}$, $\theta_{*}\geq\theta_{\min}>0$. Suppose there exist two different solutions $(\sigma_{1},\tau_{1})$ and $(\sigma_{2},\tau_{2})$ to (13)-(14), then since $\sigma_{1},\tau_{1},\sigma_{2},\tau_{2}>0$, $(\sigma_{1},\sigma_{1}/\tau_{1})$ and $(\sigma_{2},\sigma_{2}/\tau_{2})$ are two different solutions to (59)-(60), leading to a contradiction. This concludes our proof.

V-C *Asymptotic Variable Selection Performance *

In this section, our goal is to prove Proposition 2. We first prove the convergence of $R_{0}^{(p)}$.

V-C1 Probabilistic upper bound of $R_{0}^{(p)}$

To prove the convergence of $R_{0}^{(p)}$, the first step is to establish the following probabilistic upper bound.

Lemma 3

For any $\varepsilon>0$, R_{0}^{(p)}\leq\mathbb{P}\big{(}\eta(Y_{*})=0\big{)}+\varepsilon, with probability approaching 1, as $p\to\infty$.

The proof of Lemma 3 is given in Appendix -E, which uses a standard approximation argument (see e.g., the proof of Lemma 2.2 (iii) and (v) in [53]).

V-C2 Probabilistic lower bound of $R_{0}^{(p)}$

The second step is to prove the following matching probabilistic lower bound for $R_{0}^{(p)}$:

Lemma 4

Under the same setting as Proposition 2, for any $\varepsilon>0$,

[TABLE]

with probability approaching 1, as $p\to\infty$.

The proof of Lemma 4, which can be found in Appendix -F, is the mostly technically involved part, so we provide more detailed explanations here.

A key strategy we adopt is using the vector

[TABLE]

as an indicator of zero coordinates of $\widehat{\boldsymbol{\beta}}$ . To give the formal statements, we need to first introduce the notion of majorization.

Definition 2

For two vectors $\boldsymbol{a},\boldsymbol{b}\in\mathbb{R}^{p}$, we say $\boldsymbol{a}$ is majorized by $\boldsymbol{b}$ (denoted as $\boldsymbol{a}\prec\boldsymbol{b}$), if for any $j\in[p]$, $\sum_{i=j}^{p}|a|_{(j)}\leq\sum_{i=j}^{p}|b|_{(j)}.$ On the other hand, we say $\boldsymbol{a}$ is strictly majorized by $\boldsymbol{b}$ (denoted as $\boldsymbol{a}\prec_{S}\boldsymbol{b}$), if for any $j\in[p]$, $\sum_{i=j}^{p}|a|_{(j)}<\sum_{i=j}^{p}|b|_{(j)}.$

Denote $|\boldsymbol{x}|_{(1:k)}$ as a vector formed by the largest $k$ components of $|\boldsymbol{x}|$. Let us call $|\boldsymbol{x}|_{(1:k)}$ as the $k$-dominant subvector of $\boldsymbol{x}$. The following is the key lemma for establishing the probabilistic lower bound of $R_{0}^{(p)}$.

Lemma 5

For the optimization problem (2), suppose for some $k\in[p]$, $|\hat{\boldsymbol{s}}|_{(1:k)}\prec_{S}\boldsymbol{\lambda}_{1:k}$ , where $\hat{\boldsymbol{s}}=\boldsymbol{A}^{\top}(\boldsymbol{y}-\boldsymbol{A}\widehat{\boldsymbol{\beta}}).$ Then we have $|\widehat{\boldsymbol{\beta}}|_{(1:k)}=\boldsymbol{0}_{k}$.

The proof of Lemma 5 can be found in Appendix -G. This characterization transform the original problem of searching zero coordinates of $\widehat{\boldsymbol{\beta}}$ into a new problem of discovering whether there is a strict majorization relation between $k$-dominant subvectors of $\hat{\boldsymbol{s}}$ and $\boldsymbol{\lambda}$. The nice thing making this strategy work is that the majorization relation between two vectors is fully captured by their empirical distributions. Besides, in our setting, the empirical distributions $\mu_{\boldsymbol{\lambda}}$ and $\mu_{\hat{\boldsymbol{s}}}$ both have simple limits: by our assumption, $\mu_{\boldsymbol{\lambda}}\to\mu_{\Lambda}$ and in Proposition 6 of Appendix -I, we show $\mu_{\hat{\boldsymbol{s}}}\to\mu_{\hat{S}}$, with $\mu_{\hat{S}}$ being the law of $\frac{Y_{*}-\eta(Y_{*})}{\tau_{*}}$.

A major part of proof of Lemma 4 is to show if condition (R.1) is satisfied and \mathbb{P}\big{(}\eta(Y_{*})=0\big{)}>0, then for k=\lfloor p[\mathbb{P}\big{(}\eta(Y_{*})=0\big{)}-\varepsilon]\rfloor, where $\varepsilon>0$ can be arbitrarily small, we have $|\hat{\boldsymbol{s}}|_{(1:k)}\prec_{S}\boldsymbol{\lambda}_{(1:k)}$ with probability approaching 1, as $p\to\infty$. Then an application of Lemma 5 will give us the desired probabilistic lower bound for $R_{0}^{(p)}$ shown in Lemma 4.

Remark 10

Let us briefly explain why $\boldsymbol{s}$ in (62) is related with the zero coordinates of $\widehat{\boldsymbol{\beta}}$. By the first order condition of (2), we can get $\hat{\boldsymbol{s}}\in\partial J_{\boldsymbol{\lambda}}(\widehat{\boldsymbol{\beta}})$, i.e., $\boldsymbol{s}$ defined in (62) is a subgradient of $J_{\boldsymbol{\lambda}}(\boldsymbol{x})$ at $\boldsymbol{x}=\widehat{\boldsymbol{\beta}}$. For non-smooth regularizer like $J_{\boldsymbol{\lambda}}$, the subgradient $\partial J_{\boldsymbol{\lambda}}$ at $\boldsymbol{x}$ can reveal some information for detecting the zero coordinates of $\boldsymbol{x}$. A simple example is LASSO: $J_{\boldsymbol{\lambda}}(\boldsymbol{x})=\lambda\|\boldsymbol{x}\|_{1}$. In this case, we have $x_{i}=0$ as long as $[\partial J_{\boldsymbol{\lambda}}(\boldsymbol{x})]_{i}\in[0,\lambda)$. This identity is used in [50] to obtain the limiting sparsity level of LASSO estimator. Here, we extend this idea to SLOPE estimator, while a key difference is that unlike the LASSO case, the zero coordinates are not determined locally: whether $x_{i}=0$ or not is not completely determined by $[\partial J_{\boldsymbol{\lambda}}(\boldsymbol{x})]_{i}$. This is mainly a consequence of non-separability of $J_{\boldsymbol{\lambda}}(\boldsymbol{x})$.

Remark 11

We can now explain where condition (R.1) originates from. By Lemma 5, we know a sufficient condition for R_{0}^{(p)}/p\geq\mathbb{P}\big{(}\eta(Y_{*})=0\big{)} is that $|\hat{\boldsymbol{s}}|_{(1:k)}\prec_{S}\boldsymbol{\lambda}_{(1:k)}$ holds for some $k$ satisfying k/p\approx\mathbb{P}\big{(}\eta(Y_{*})=0\big{)}:=q_{0}^{*}. In the asymptotic limit, this translate into the following limiting form: for any $t\in[0,q_{0}^{*})$,

[TABLE]

In fact, (63) is exactly (R.1), since for any $u\in[0,q_{0}^{*}]$ we have $F_{|Y_{*}-\eta(Y_{*})|/\tau_{*}}^{-1}(u)=F_{|Y_{*}|/\tau_{*}}^{-1}(u)$.

After combining the probabilistic upper and lower bounds, we conclude that R_{0}^{(p)}\overset{\mathbb{P}}{\rightarrow}\mathbb{P}\big{(}\eta(Y_{*})=0\big{)}.

V-C3 Convergence of $V^{(p)}$

Finally, we prove the convergence of $V^{(p)}$. We have the following lemma, which shows V^{(p)}\overset{\mathbb{P}}{\rightarrow}\mathbb{P}\big{(}\eta(Y_{*})\neq 0,B=0\big{)} can be implied by R_{0}^{(p)}\overset{\mathbb{P}}{\rightarrow}\mathbb{P}\big{(}\eta(Y_{*})=0\big{)}.

Lemma 6

For any $\varepsilon>0$,

[TABLE]

with probability approaching 1 as $p\to\infty$.

The proof of Lemma 6 can be found in Appendix -H. Since the convergence of $R_{0}^{(p)}$ has been established, we finish our proof.

V-D Optimal Estimation

In this section, we prove the fundamental estimation performance of SLOPE, as stated in Proposition 4. The proof of part (a) and (b), which justifies the uniqueness of $f_{\sigma}$ and the existence of $\sigma_{0}$, can be found in Lemma 18 and Lemma 19 in Appendix -M. Here we focus on proving part (c), which is the core part of Proposition 4.

From discussions before, finding the minimum MSE is equivalent to solving (24). Indeed, we have

[TABLE]

where $\sigma_{\text{opt}}$ is the optimal value of (24), i.e., $\sigma_{\text{opt}}\overset{\text{def}}{=}\inf_{\mu_{\Lambda}\in\mathcal{P}_{\Lambda}}\sigma_{*}$.

V-D1 A reformulation of $\inf_{\mu_{\Lambda}\in\mathcal{P}_{\Lambda}}\sigma_{*}$

We start by noting that $\sigma_{\text{opt}}$ can be equivalently expressed as:

[TABLE]

where

[TABLE]

Geometrically, computing $\sigma_{\text{opt}}$ is equivalent to searching for the leftmost point in $\mathcal{D}_{L}$, which is the set of all realizable $(\sigma,\tau)$ pair. However, characterizing $\mathcal{D}_{L}$ is difficult, since it is determined in a convoluted way via (13)-(14). To simplify, consider instead the following equation of $(f,\sigma,\tau)$

[TABLE]

where $(f,\sigma,\tau)\in\mathcal{I}\times\mathbb{R}_{>0}\times\mathbb{R}_{>0}$ and $H\sim\mathcal{N}(0,1)$ is independent of $B\sim\mu_{B}$. Let us emphasize that although (66)-(67) has a similar form as (13)-(14), a key difference is that unlike $\eta$ in (13)-(14), $f$ in (66)-(67) is not dependent on other parameters such as $(B,H,\sigma,\tau)$.

Now define the following set of $(\sigma,\tau)$:

[TABLE]

A key step of our proof is to show $\mathcal{D}_{L}=\mathcal{D}_{F}$. This can be done as follows. Clearly, we have $\mathcal{D}_{L}\subseteq\mathcal{D}_{F}$, since $\eta\in\mathcal{I}$. To prove $\mathcal{D}_{F}\subseteq\mathcal{D}_{L}$, we need to utilize Proposition 3. Suppose $(\sigma,\tau)\in\mathcal{D}_{F}$ and let $(f,\sigma,\tau)$ be the corresponding solution of (66)-(67). If $\delta>1$, we have $\mathcal{P}_{\Lambda}=\mathcal{P}_{2}(\mathbb{R})$ and by Proposition 3 we can take $\Lambda\sim\frac{|Y|-f(|Y|)}{\tau}\in\mathcal{P}_{2}(\mathbb{R})$ so that $\eta(\cdot;\mu_{Y},\mu_{\tau\Lambda})=f$; if $\delta\leq 1$, then from (67) we know $f(y)\neq y$ and \mathbb{P}\big{(}\frac{|Y|-f(|Y|)}{\tau}\neq 0\big{)}>0, so $\frac{|Y|-f(|Y|)}{\tau}\in\mathcal{P}_{\Lambda}$ and we can still take $\Lambda\sim\frac{|Y|-f(|Y|)}{\tau}$ which gives us $\eta(\cdot;\mu_{Y},\mu_{\tau\Lambda})=f$. This means $(\sigma,\tau)\in\mathcal{D}_{L}$. As a result, we conclude that $\mathcal{D}_{F}\subseteq\mathcal{D}_{L}$ and thus $\mathcal{D}_{L}=\mathcal{D}_{F}$. Then substituting $\mathcal{D}_{L}=\mathcal{D}_{F}$ into (65), we get the following reformulation of $\sigma_{\text{opt}}$:

[TABLE]

V-D2 Lower Bound of MSE

Note that any $(f,\sigma)\in\mathcal{I}\times\mathbb{R}_{>0}$ satisfying (66)-(67) for some $\tau>0$, should also satisfy

[TABLE]

Therefore, if we consider the following set of $\sigma$:

[TABLE]

then from (68) we have

[TABLE]

Compared with $\sigma_{\text{opt}}$, the lower bound $\inf\mathcal{A}$ in (71) is easier to obtain, since the variable $\tau$ is dropped. In Lemma 19 in Appendix -M, we show that $\inf\mathcal{A}=\sigma_{0}$. Therefore, $\sigma_{\text{opt}}\geq\sigma_{0}$. Together with (64), we prove (30).

V-D3 Reaching the Lower Bound

We now show lower bound $\sigma_{\text{opt}}\geq\sigma_{0}$ is tight, if \delta^{-1}\mathbb{E}\big{[}f_{\sigma_{0}}^{\prime}(B+\sigma_{0}H)\big{]}<1. Recall that $f_{\sigma_{0}}$ is the unique optimal solution of (28) when $\sigma=\sigma_{0}$ and $(f_{\sigma_{0}},\sigma_{0})$ satisfies (69). Let \tau_{0}=\big{[}1-\delta^{-1}\mathbb{E}f_{\sigma_{0}}^{\prime}(B+\sigma_{0}H)\big{]}^{-1}. It is not hard to see when \delta^{-1}\mathbb{E}\big{[}f_{\sigma_{0}}^{\prime}(B+\sigma_{0}H)\big{]}<1, $\tau_{0}\in(0,\infty)$ and $(f_{\sigma_{0}},\sigma_{0},\tau_{0})$ is a solution of (66)-(67). This indicates $(\sigma_{0},\tau_{0})\in\mathcal{D}_{F}$ and thus from (68) we have $\sigma_{\text{opt}}\leq\sigma_{0}$. Together with the lower bound $\sigma_{\text{opt}}\geq\sigma_{0}$, we get $\sigma_{\text{opt}}=\sigma_{0}$.

On the other hand, by Proposition 3 we know if $\mu_{\Lambda}$ is taken as the law of \frac{1}{\tau_{0}}\big{(}|Y_{0}|-f_{\sigma_{0}}(|Y_{0}|)\big{)}, then $f_{\sigma_{0}}(y)=\eta(y;\mu_{Y_{0}},\mu_{\tau_{0}\Lambda})$. Since $(f_{\sigma_{0}},\sigma_{0},\tau_{0})$ is a solution of equation (66)-(67), we know $(\sigma_{*},\tau_{*})=(\sigma_{0},\tau_{0})$, where $(\sigma_{*},\tau_{*})$ is the solution of fixed-point equation (13)-(14) under this choice of $\mu_{\Lambda}$. According to (15), we get $\lim_{p\to\infty}\frac{\|\widehat{\boldsymbol{\beta}}-\boldsymbol{\beta}\|_{2}^{2}}{p}=\delta(\sigma_{0}^{2}-\sigma_{w}^{2})$. This completes our proof.

V-E Optimal Variable Selection

In this section, we are going to prove Proposition 5. First, part (a) and (b) can be proved in an analogous way as in Proposition 5, which is summarized in Lemma 21. Here we focus on part (c).

V-E1 Upper bound of $\mathcal{P}(\alpha)$

Directly solving the original optimization (40) is not easy. Instead, replacing by a new objective function in (40), we will first consider the following problem:

[TABLE]

where $y_{\text{th}}^{*}=\sup_{y\geq 0}\{y\mid\eta(y;\mu_{Y_{*}},\mu_{\tau_{*}\Lambda})=0\}$. It is not hard to show for any $\alpha\in[0,1]$, we have $\overline{\mathcal{\mathcal{P}}}(\alpha)\geq\mathcal{P}(\alpha)$. This is because the constraint $\mathbb{P}(|\sigma_{*}H|\geq y_{\text{th}}^{*})\leq\alpha$ in (40) implies $y_{\text{th}}^{*}\geq\Phi^{-1}(1-\frac{\alpha}{2})\sigma_{*}$ and hence the objective function of (40) is upper bounded by that of (72) for any $\mu_{\Lambda}\in\widetilde{\mathcal{P}}_{\Lambda}$.

Problem (72) can be further simplified. By direct differentiation, one can check for any fixed $b\neq 0$ and $c\geq 0$, the function $\sigma\mapsto\mathbb{P}(|b+\sigma H|\geq c\sigma)$ is non-increasing on $\mathbb{R}_{\geq 0}$, where $H\sim\mathcal{N}(0,1)$. This then implies, by conditioning on $B$, that $\sigma\mapsto\mathbb{P}(|B+\sigma H|\geq c\sigma\mid B\neq 0)$ is non-increasing on $\mathbb{R}_{\geq 0}$ for any distribution of $B$ satisfying $\mathbb{P}(B\neq 0)>0$. Therefore, solving maximization problem in (72) is equivalent to solving the minimization problem of $\sigma_{*}$. Meanwhile, by the definition of $y_{\text{th}}^{*}$ and the fact that $\eta(y;\mu_{Y_{*}},\mu_{\tau_{*}\Lambda})\in\mathcal{I}$, we know: $\mathbb{P}(|\sigma_{*}H|\geq y_{\text{th}}^{*})\leq\alpha$ if and only if $\eta(y;\mu_{Y_{*}},\mu_{\tau_{*}\Lambda})=0$, for all $|y|\leq\Phi^{-1}(1-\frac{\alpha}{2})\sigma_{*}$. As a result, solving (72) is equivalent to solving:

[TABLE]

and $\overline{\mathcal{\mathcal{P}}}(\alpha)$ in (72) can be expressed in terms of $\sigma_{\text{opt},\alpha}$ in (73) as:

[TABLE]

So far, we arrive at the optimization problem (73), which is similar to the one that we have analyzed in the estimation setting [c.f. (24)]. Yet there are two differences: (i) a constraint on $\eta$ is added to ensure type-I error is bounded by $\alpha$, (ii) a constraint on $\mu_{\Lambda}$ is added to guarantee valid limits of Type-I error and power exist (see Proposition 2). It turns out that the strategy we used can still be applied. The results are parallel to Proposition 4 part (c) and are summarized in Lemma 22 in Appendix -N, where it is shown that

[TABLE]

and the lower bound can be achieved when $\mu_{\Lambda}=\mu_{\text{opt},\alpha}$, if \delta^{-1}\mathbb{E}\big{[}f_{\alpha}^{\prime}(Y_{0,\alpha})\big{]}<1. After combining (75) with

[TABLE]

and (39), we get (43).

V-E2 Reaching the upper bound

Now we show for any $\alpha\in[0,1]$, the upper bound (43) is tight, if \delta^{-1}\mathbb{E}\big{[}f_{\alpha}^{\prime}(Y_{0,\alpha})\big{]}<1 and $y_{0,\alpha}=\Phi^{-1}(1-\frac{\alpha}{2})\sigma_{0,\alpha}$. Also it is attained by $\mu_{\Lambda}=\mu_{\text{opt},\alpha}$. The case of $\alpha=0$ is easy. Indeed, in this case, both sides of (43) equal to 0. We just need to verify the case of $\alpha\in(0,1]$. By Lemma 22 and (76), we know it suffices to show $\mathcal{P}(\alpha)=\overline{\mathcal{\mathcal{P}}}(\alpha)$ and also $\lim_{p\to\infty}\text{Power}=\mathcal{P}(\alpha)$, when $\mu_{\Lambda}=\mu_{\text{opt},\alpha}$. We verify this in Lemma 23 in Appendix -N, which completes our proof.

VI Concluding Remarks

We have established the asymptotic characterization of SLOPE in the high-dimensional regime. Although SLOPE is a high-dimensional regularized regression method, asymptotically its statistical performance can be fully characterized by a few scalar random variables. The precise characterization enabled us to derive the fundamental performance limits of SLOPE for both estimation and variable selection settings. Also we showed how to design the optimal regularizing sequences that achieve these limits.

Finally, let us point out some generalizations of current results that worth exploring in the future.

One major technical assumption in the current paper is that the sensing matrix is generated from i.i.d. Gaussian. There are two possible ways to relax this assumption. The first one is to consider the Gaussian design with correlated columns, which is the setting analyzed in [6]. Under this scenario, SLOPE enjoys the nice properties of selecting all the variables associated with highly correlated columns. It would be interesting to derive a precise explanation for this phenomenon. The second direction is staying in the i.i.d. setting, while generalizing to other ensembles, e.g., sub-Gaussian distribution. This is to verify the so-called universality phenomenon and some works have been done in the setting where the regularizer is separable [54, 55]. It would be interesting to generalize these results to non-separable regularizers such as SLOPE. 2. 2.

The optimal designs of $\boldsymbol{\lambda}$ sequences considered in this paper are based on the assumption that the true distribution of unknown signal is known. The natural question is: can we design $\boldsymbol{\lambda}$ sequences without (or just with partial) such prior knowledge? One related problem is designing a regularizing sequence such that the false discovery rate is always controlled under a given level. In this setting, the realistic assumption is that we do not know the sparsity of underlying signal. For this purpose, a design of $\boldsymbol{\lambda}$ is proposed in [9] based on some qualitative insights. It would be nice to have quantitative results utilizing the exact characterizations derived here. 3. 3.

From numerical simulations, we can find that in several cases, the performance of practical $\boldsymbol{\lambda}$ sequences such as LASSO and BHq is comparable to the optimal performance. Is it possible that the optimal performance of SLOPE can actually be approximately achieved, when we are restricted to certain sub-classes of regularizing sequences? A key step is establishing some easy-to-evaluate bounds for the performance gap between practical and optimal sequences. One benefit of using practical sequences is that we can apply some purely data-dependent methods such as cross-validation to search for the optimal tuning parameter. Note that since general $\boldsymbol{\lambda}$ sequence includes order $\mathcal{O}(p)$ parameters, the grid search approach that is usually used in data-dependent method is not plausible here.

-A Proof of Lemma 1

First assume $0\leq y_{1}\leq\cdots\leq y_{p}$. Denote $\hat{\boldsymbol{y}}:=\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{y})$. Then consider the linear interpolation of the points $\{(y_{i},\hat{y}_{i})\}_{i=0}^{p}$, where $(y_{0},\hat{y}_{0})=(0,0)$:

[TABLE]

By Fact 1 (iii), we know $g_{p}^{+}(y)$ is non-decreasing and 1-Lipschitz continuous on $\mathbb{R}_{\geq 0}$.

For general $\boldsymbol{y}$, we first obtain the linear interpolation $g_{p}^{+}(y)$ of the points $\{(|y|_{(i)},|\hat{y}|_{(i)})\}_{i=0}^{p}$ as above. Then $g_{p}(y)$ can be constructed as follows:

[TABLE]

Clearly, such $g_{p}(y)$ is an odd, non-decreasing and 1-Lipschitz function. Also by Fact 1 (i) and (ii), one can easily check $g_{p}(y_{i})=\hat{y}_{i}$, for all $i\in[p]$. This finishes the proof.

-B Proof of Lemma 2

For notational simplicity, denote $\mathcal{M}_{\boldsymbol{\lambda}}(\boldsymbol{y}):=\mathcal{M}_{\boldsymbol{\lambda}}(\boldsymbol{y};1)$. Let $g_{p}^{*}(\boldsymbol{y})$ be any minimizer of (45). Since $\mathcal{M}_{\boldsymbol{\lambda}}(\boldsymbol{y})$ is the minimum value of (5),

[TABLE]

Next, we show $p\mathcal{M}_{\boldsymbol{\lambda}}^{*}(\boldsymbol{y})\leq\mathcal{M}_{\boldsymbol{\lambda}}(\boldsymbol{y})$. Given Lemma 1, this is immediate. Indeed,

[TABLE]

where $g_{p}$ is the function we construct in Lemma 1, which satisfies $g_{p}\in\mathcal{I}$ and $g_{p}(\boldsymbol{y})=\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{y})$. Combining (78) and (79), we get $\mathcal{M}_{\boldsymbol{\lambda}}^{*}(\boldsymbol{y})=\frac{\mathcal{M}_{\boldsymbol{\lambda}}(\boldsymbol{y})}{p}$.

Substituting $\mathcal{M}_{\boldsymbol{\lambda}}^{*}(\boldsymbol{y})=\frac{\mathcal{M}_{\boldsymbol{\lambda}}(\boldsymbol{y})}{p}$ into (78), we have for any minimizer $g_{p}^{*}$ of (45), $g_{p}^{*}(\boldsymbol{y})$ is also a minimizer of (5). Since $\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{y})$ is the unique minimizer of (5), we know any minimizer of (45) should satisfy: $g_{p}^{*}(\boldsymbol{y})=\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{y})$.

-C Auxiliary Results for Proving Proposition 1

Lemma 7

Suppose $\{\boldsymbol{y}^{(p)}\}_{p\in\mathbb{Z}^{+}}$ and $\{\boldsymbol{\lambda}^{(p)}\}_{p\in\mathbb{Z}^{+}}$ are converging sequences with limiting measure $\mu_{Y}$ and $\mu_{\Lambda}$. Then

[TABLE]

where $L(g)$ and $L_{p}(g)$ are defined in (46) and (45).

Proof:

The first step is to establish the following uniform convergence of a class of Pseudo-Lipschitz functions. Let $\Psi$ be the set of all functions $\psi:\mathbb{R}\to\mathbb{R}$ satisfying: $\psi(0)=0$ and $|\psi(x)-\psi(y)|\leq(1+|x|+|y|)|x-y|$ for any $x,y\in\mathbb{R}$. Then

[TABLE]

To prove (81), first for any $\psi\in\Psi$ consider the truncation:

[TABLE]

where $A>0$ is a constant. It is easy to check $\hat{\psi}_{A}(y)$ is $(1+2A)$-Lipschitz continuous, so

[TABLE]

where (a) follows from Kantonovich duality theorem ([56, Theorem 1.3]) and (b) follows from Holder’s inequality. Therefore,

[TABLE]

For any $\varepsilon>0$, each term on the RHS of (82) can be bounded as follows. Since $(\mathbb{E}_{\mu_{Y}}|Y|)^{2}\leq\mathbb{E}_{\mu_{Y}}Y^{2}<\infty$, by DCT there always exists $A>0$ such that $\mathbb{E}_{\mu_{Y}}[\mathbb{I}_{|Y|\geq A}(Y^{2}+|Y|)]\leq\frac{\varepsilon}{4}$. On the other hand, for any given $A>0$ and $\varepsilon>0$, there always exists $p_{0}\in\mathbb{N}$ such that for any $p\geq p_{0}$, (i) $W_{2}(\mu_{\boldsymbol{y}},\mu_{Y})\leq\frac{\varepsilon}{4(1+2A)}$, since $W_{2}(\mu_{\boldsymbol{y}},\mu_{Y})\to 0$ and (ii) $\mathbb{E}_{\mu_{\boldsymbol{y}}}[\mathbb{I}_{|Y|\geq A}(Y^{2}+|Y|)]\leq\mathbb{E}_{\mu_{Y}}[\mathbb{I}_{|Y|\geq A}(Y^{2}+|Y|)]+\frac{\varepsilon}{4}$ by Theorem 7.12 (iv) in [56]. Note that the RHS of (82) does not depend on $\psi$, so for any $\varepsilon>0$, there exists $p_{0}\in\mathbb{N}$ such that for any $p\geq p_{0}$, \sup_{\psi\in\Psi}\big{|}\mathbb{E}_{\mu_{\boldsymbol{y}}}\psi(Y)-\mathbb{E}_{\mu_{Y}}\psi(Y)\big{|}\leq\varepsilon. Therefore, (81) is proved.

We are now ready to show (80). Recalling the definitions of $L(g)$ and $L_{p}(g)$ in (46) and (45), we have

[TABLE]

Therefore, it remains to control each term on the RHS of (83). The first term can be handled by using (81), since $y\mapsto[y-g(y)]^{2}$ belongs to $\Psi$ for $g\in\mathcal{I}$; the second term can be controlled as : $\text{Term II}\leq W_{2}(\mu_{\boldsymbol{\lambda}},\mu_{\Lambda})\sqrt{\mathbb{E}_{\mu_{Y}}Y^{2}}$ using (86) and Cauchy-Swartz inequality; similarly for the third term, we have

[TABLE]

where the last inequality follows from the definition of Wasserstein distance and the Lipschitz continuity of $g$:

[TABLE]

Substituting the above bounds back to (83) and using the assumption that W_{2}(\mu_{\boldsymbol{\lambda}},\mu_{\Lambda}),W_{2}\big{(}\mu_{\boldsymbol{y}},\mu_{Y}\big{)}\to 0, we obtain the desired results. ∎

Lemma 8

The optimization problem (6) has an optimal solution and it is unique (up to a set of measure 0 with respect to $\mu_{Y}$).

Proof:

Without loss of generality, we assume $\tau=1$. The objective function $L(g)$ of (6) is defined on the following $L^{2}$ space:

[TABLE]

where $\|g\|_{\mu_{Y}}\overset{\text{def}}{=}[\mathbb{E}_{\mu_{Y}}g^{2}(Y)]^{1/2}$. It is known that in $L^{2}$ space (and more generally in all normed linear spaces), the convention is to work with equivalence class of functions [57, p.135-136]. The equivalence class of a function $f\in\mathcal{H}_{\mu_{Y}}$, denoted as $[f]$, is the collection of all functions $g\in\mathcal{H}_{\mu_{Y}}$ satisfying $\|g-f\|_{\mu_{Y}}=0$. As a notational convention, we will write $[f]$ as $f$, and the set $\{[f]:f\in\mathcal{I}\}$ as $\mathcal{I}$. Also $\|g-f\|_{\mu_{Y}}=0$ will be denoted as $g=f$.

We first show $L(g)$ is 1-strongly convex on $\mathcal{H}_{\mu_{Y}}$, i.e., for any $g_{1},g_{2}\in\mathcal{H}_{\mu_{Y}}$,

[TABLE]

First, for any $\theta\in[0,1]$,

[TABLE]

which implies that $L_{1}(g):=\int_{0}^{1}F_{\Lambda}^{-1}(u)F_{|g(Y)|}^{-1}(u)du$ is convex. Also, it is not hard to check $L_{2}(g):=\frac{1}{2}\mathbb{E}_{\mu}[Y-g(Y)]^{2}$ is 1-strongly convex by definition (85). Then the strong convexity of $L(g)$ follows, since $L(g)=L_{1}(g)+L_{2}(g)$. On the other hand, we can show $L(g)$ is continuous on $\mathcal{H}_{\mu_{Y}}$. Indeed,

[TABLE]

Since $\|y\|_{\mu_{Y}},\|g_{1}\|_{\mu_{Y}},\|g_{2}\|_{\mu_{Y}}<\infty$, we conclude that $L(g)$ is continuous.

Next we are going to show the set $\mathcal{I}$ is convex, bounded and closed in $\mathcal{H}_{\mu_{Y}}$. The convexity can be directly checked by definition. Choose any $g_{1},g_{2}\in\mathcal{I}$. Then there exists $S\subseteq\mathbb{R}$ with $\mu_{Y}(S)=1$ such that for any $y_{1},y_{2}\in S$, $y_{1}\leq y_{2}$ and any $y\in S$, we have $0\leq g_{i}(y_{2})-g_{i}(y_{1})\leq y_{2}-y_{1}$ and $g_{i}(y)=-g_{i}(-y)$, where $i=1,2$. Then for any $\theta\in[0,1]$, function $\theta g_{1}+(1-\theta)g_{2}$ also satisfy (i) and (ii) on $S$, so $\theta g_{1}+(1-\theta)g_{2}\in\mathcal{I}$. The boundedness directly follows from the fact that for any $g\in\mathcal{I}$, $g(y)\leq|y|$ on some $S\subseteq\mathbb{R}$ with $\mu_{Y}(S)=1$ . To show closedness, suppose $g_{k}(y)\in\mathcal{I},k=1,2,\ldots$ is a sequence of functions that converge to some $g(y)\in\mathcal{H}_{\mu_{Y}}$. Then by Riesz-Fischer Theorem, there exists a sub-sequence of $\{g_{k}(y)\}_{k\in\mathbb{Z}^{+}}$ that converges point-wise to $g(y)$ on some $S\subseteq\mathbb{R}$ with $\mu_{Y}(S)=1$. By this $\mu_{Y}$-almost everywhere convergence of $g_{k}(y)$ to $g(y)$, we know there exists some $S^{\prime}\subseteq\mathbb{R}$ with $\mu_{Y}(S^{\prime})=1$, such that for any $y_{1},y_{2}\in S^{\prime}$,$y_{1}\leq y_{2}$ and any $y\in S^{\prime}$, it holds that $0\leq g(y_{2})-g(y_{1})\leq y_{2}-y_{1}$ and $g(y)=-g(-y)$. Therefore, $g(y)\in\mathcal{I}$ and thus $\mathcal{I}$ is closed.

The final step is to apply Theorem 17 in [57, Chap. 8] to conclude that (6) has an optimal solution $g^{*}\in\mathcal{I}$. Also the uniqueness of $g^{*}$ can be easily checked by the strong convexity of $L(g)$. Suppose there exists two different optimal solutions, $g_{1}^{*},g_{2}^{*}$ with $L(g_{1}^{*})=L(g_{2}^{*})$ and $g_{1}^{*}\neq g_{2}^{*}$. Then by (85), for $g=\frac{g_{1}+g_{2}}{2}\in\mathcal{I}$ we have $L(g)<L(g_{1}^{*})=L(g_{2}^{*})$, which leads to a contradiction. ∎

The following result provides the explicit formula for calculating Wasserstein-2 distance between probability measure on $\mathbb{R}$. Readers can find a proof in Theorem 2.18 in [56].

Lemma 9

Suppose $\mu_{1},\mu_{2}\in\mathcal{P}_{2}(\mathbb{R})$ and the corresponding quantile functions are $F_{1}^{-1}$ and $F_{2}^{-1}$. Then

[TABLE]

-D Auxiliary Results for Proving Theorem 1

In this section, we prove three auxiliary lemmas used in the proof of Theorem 1.

The first two results are on the asymptotic properties of auxiliary problem (52). To state these asymptotic results, similar as Theorem 1, we will consider a sequence of auxiliary problems described by the instances $\{\boldsymbol{g}^{(p)},\boldsymbol{h}^{(p)},\boldsymbol{\beta}^{(p)},\boldsymbol{w}^{(p)},\boldsymbol{\lambda}^{(p)}\}_{p\in\mathbb{Z}^{+}}$. They satisfy the following: (i) $\boldsymbol{g}^{(p)}\sim\mathcal{\mathcal{N}}(\boldsymbol{0},\boldsymbol{I}_{n})$, $\boldsymbol{h}^{(p)}\sim\mathcal{\mathcal{N}}(\boldsymbol{0},\boldsymbol{I}_{p})$, $p\in\mathbb{Z}^{+}$ are all independent, (ii)$\{\boldsymbol{\beta}^{(p)}\}_{p\in\mathbb{Z}^{+}}$, $\{\boldsymbol{w}^{(p)}\}_{p\in\mathbb{Z}^{+}}$, $\{\boldsymbol{\lambda}^{(p)}\}_{p\in\mathbb{Z}^{+}}$ are the same converging sequences as in Theorem 1. Here the requirement that $\{\boldsymbol{g}^{(p)}\}_{p\in\mathbb{Z}^{+}}$ and $\{\boldsymbol{h}^{(p)}\}_{p\in\mathbb{Z}^{+}}$ are independent is not completely necessary, since we are only aiming for results regarding convergence in probability. The independence assumption simply allows us to directly apply some results obtained in Appendix -K.

The first lemma is about the minimum value and the minimizer of $L(\boldsymbol{v})$ over a bounded Euclidean ball. Recall that

[TABLE]

Lemma 10

Let $\Psi_{*}$ and $(\sigma_{*},\theta_{*})$ be the optimal value and solution of the minimax problem in (55) and $\theta_{\min}>0$ be the lower bound of $\theta_{*}$ obtained in Lemma 14. For any $\varepsilon>0$ and $K\geq\tfrac{\sigma_{*}}{\sqrt{\delta}}+\tfrac{\theta_{\min}}{4}$, we have

[TABLE]

and

[TABLE]

where $\gamma=\frac{\theta_{\min}\sigma_{w}^{2}}{4(K^{2}+\sigma_{w}^{2})^{3/2}}$ and

[TABLE]

Here in (90), $\eta(\cdot):=\eta(\boldsymbol{\beta}+\sigma_{*}\boldsymbol{h};\mu_{Y_{*}},\mu_{\sigma_{*}\Lambda/\theta_{*}})$ and $Y_{*}=B+\sigma_{*}H$, with $H\sim\mathcal{N}(0,1)$ independent of $B\sim\mu_{B}$.

Proof:

We follow the proof of Proposition B.2 in [50]. First introduce the event $\mathcal{A}=\bigcap_{i=1}^{5}\mathcal{A}_{i}$, where

[TABLE]

with $\varepsilon>0$ and $\varsigma\in(0,\tfrac{1}{2})$. In (91), $\mathcal{F}_{p}$ and $\mathcal{F}$ are the same as in (160) and (163) and $\widetilde{L}(\boldsymbol{v})$ is defined as:

[TABLE]

Based on the event $\mathcal{A}$, our subsequent analysis will become fully deterministic: we will condition on fixed $\boldsymbol{h}$ and $\boldsymbol{g}$ in $\mathcal{A}$. Before doing so, let us first show each of the events $\mathcal{A}_{1}\sim\mathcal{A}_{5}$ occurs with probability approaching 1 as $p\to\infty$, so $\mathbb{P}(\mathcal{A})\to 1$. This will ensure all the results obtained by conditioning on $\mathcal{A}$ hold with probability approaching 1.

$\mathcal{A}_{1}$: By the law of large number and the fact that $\{\boldsymbol{w}\}_{p\in\mathbb{N}}$ is a converging sequence with limiting variance $\sigma_{w}^{2}>0$, it is not hard to show $\mathbb{P}(\mathcal{A}_{1})\to 1$.

$\mathcal{A}_{2}$: From (179), we have

[TABLE]

Then together with (151), we get $\frac{\|\mathring{\boldsymbol{v}}\|^{2}}{p}\overset{a.s.}{\rightarrow}(\sigma_{*})^{2}-\sigma_{w}^{2}$. Therefore, $\mathbb{P}(\mathcal{A}_{2})\to 1$, since $\sigma_{w}^{2}>0$ by assumption.

$\mathcal{A}_{3}$: From (179) and (181), we can get

[TABLE]

where the last step follows from (151). From Lemma 14, there exists $\theta_{\min}>0$ such that $\theta_{*}\geq\theta_{\min}$. Therefore, $\mathbb{P}(\mathcal{A}_{3})\to 1$.

$\mathcal{A}_{4}$: From the definition of $\mathcal{F}(\sigma,\theta)$ in (163),

[TABLE]

where in the last step we use (151). Then substituting (95) into (55), we get

[TABLE]

On the other hand,

[TABLE]

where we use (94). From (96) and (97), we have $\widetilde{L}(\mathring{\boldsymbol{v}})\overset{a.s.}{\rightarrow}\Psi_{*}$. Therefore, $\mathbb{P}(\mathcal{A}_{4})\to 1$ for any $\varepsilon>0$.

$\mathcal{A}_{5}$: From (161) and strong law of large number for triangular array [58, Theorem 2.1], we have $\mathcal{F}_{p}(\sigma,\theta_{*})-\tfrac{\sigma\theta_{*}\|\boldsymbol{h}\|^{2}}{2p}\overset{a.s.}{\rightarrow}\mathcal{F}(\sigma,\theta_{*})-\tfrac{\sigma\theta_{*}}{2}$ for any $\sigma\in[\sigma_{w},\sqrt{\sigma_{w}^{2}+K^{2}}]$. From (167) in Lemma 16, we know $\mathcal{F}(\cdot,\theta_{*})$ is Lipschitz continuous on $[\sigma_{w},\sqrt{\sigma_{w}^{2}+K^{2}}]$. This indicates $\sigma\mapsto\mathcal{F}(\sigma,\theta_{*})-\tfrac{\sigma\theta_{*}}{2}$ is also Lipschitz continuous on $[\sigma_{w},\sqrt{\sigma_{w}^{2}+K^{2}}]$. On the other hand, in the proof of Lemma 16 [line below (174)], we show $\mathcal{F}_{p}(\cdot,\theta_{*})$ is continuously differentiable on $[\sigma_{w},\infty)$ with derivative satisfying $\left|\frac{\partial\mathcal{F}_{p}(\sigma,\theta_{*})}{\partial\sigma}\right|\leq\frac{3\theta_{*}\|\boldsymbol{\beta}\|^{2}+\sigma^{2}(2+\theta_{*})\|\boldsymbol{h}\|^{2}}{\sigma^{2}p}$. Then by [58, Theorem 2.1] again and the fact that $\{\boldsymbol{\beta}\}_{p\in\mathbb{Z}^{+}}$ is a converging sequence, we have almost surely $\limsup_{p\to\infty}\left|\frac{\partial\mathcal{F}_{p}(\sigma,\theta_{*})}{\partial\sigma}-\tfrac{\theta_{*}\|\boldsymbol{h}\|^{2}}{2p}\right|\leq C$ for any $\sigma\in[\sigma_{w},\sqrt{\sigma_{w}^{2}+K^{2}}]$, where $C>0$ is some constant. This indicates that almost surely, $\sigma\mapsto\frac{\partial\mathcal{F}_{p}(\sigma,\theta_{*})}{\partial\sigma}-\tfrac{\theta_{*}\|\boldsymbol{h}\|^{2}}{2p}$ is $C$-Lipschitz continuous on $[\sigma_{w},\sqrt{\sigma_{w}^{2}+K^{2}}]$ for any large enough $p$. Then by the same epsilon net argument as in the proof of Lemma 16, we can show

[TABLE]

Therefore, $\mathbb{P}(\mathcal{A}_{5})\to 1$ for any $\varepsilon>0$.

Now we are ready to start the deterministic analysis conditioned on the event $\mathcal{A}$. It is more convenient to work with $\widetilde{L}(\boldsymbol{v})$ than $L(\boldsymbol{v})$, since it is locally strongly convex (the precise meaning will be given below). In the sequel, we will start by studying the limiting properties of $\widetilde{L}(\boldsymbol{v})$ and then associate them to $L(\boldsymbol{v})$, by showing $L(\boldsymbol{v})$ can be well-approximated by $\widetilde{L}(\boldsymbol{v})$ as $p\to\infty$. For $\boldsymbol{v}\in\mathcal{B}_{\sqrt{n}K}$, $\widetilde{L}(\boldsymbol{v})$ can be equivalently written as:

[TABLE]

Therefore,

[TABLE]

where (a) follows from (98) and (160), (b) is due to $\mathcal{A}_{5}$ and (c) is due to $\mathcal{A}_{4}$. Besides, since $\tfrac{\|\mathring{\boldsymbol{v}}\|_{2}}{\sqrt{n}}\leq\tfrac{\sigma_{*}}{\sqrt{\delta}}$ under $\mathcal{A}_{2}$ and $\tfrac{\sigma_{*}}{\sqrt{\delta}}\leq K-\frac{\theta_{\min}}{4}$ by assumption, we have $\tfrac{\|\mathring{\boldsymbol{v}}\|_{2}}{\sqrt{n}}\leq K$ and thus $\min_{\boldsymbol{v}\in\mathcal{B}_{\sqrt{n}K}}\widetilde{L}(\boldsymbol{v})\leq\widetilde{L}(\mathring{\boldsymbol{v}})\leq\Psi_{*}+\varepsilon.$ Therefore, combining it with (99) yields

[TABLE]

On the other hand, $\boldsymbol{v}\mapsto\sqrt{\tfrac{\|\boldsymbol{v}\|^{2}}{n}+\sigma_{w}^{2}}-\tfrac{\boldsymbol{h}^{\top}\boldsymbol{v}}{n}$ is $\tfrac{1}{\sqrt{n}}(\sqrt{\tfrac{3}{2\delta}}+1)$-Lipschitz continuous under $\mathcal{A}_{1}$ and $\sqrt{\tfrac{\|\mathring{\boldsymbol{v}}\|^{2}}{n}+\sigma_{w}^{2}}-\tfrac{\boldsymbol{h}^{\top}\mathring{\boldsymbol{v}}}{n}\geq\tfrac{\theta_{\min}}{2}$ under $\mathcal{A}_{3}$. Therefore, for $r=(\sqrt{\tfrac{3}{2\delta}}+1)^{-1}\tfrac{\theta_{\min}}{4}$, $\sqrt{\tfrac{\|\boldsymbol{v}\|^{2}}{n}+\sigma_{w}^{2}}-\tfrac{\boldsymbol{h}^{\top}\boldsymbol{v}}{n}\geq\tfrac{\theta_{\min}}{4}$ for all $\boldsymbol{v}\in\mathcal{B}_{\sqrt{n}r}(\mathring{\boldsymbol{v}})$. Then using Lemma F.14 in [50] and $\tfrac{\|\mathring{\boldsymbol{v}}\|_{2}}{\sqrt{n}}+r\leq\tfrac{\sigma_{*}}{\sqrt{n}}+r<K$ (hence $\mathcal{B}_{\sqrt{n}r}(\mathring{\boldsymbol{v}})\subset\mathcal{B}_{\sqrt{n}K}$) under $\mathcal{A}_{2}$, we can show $\widetilde{L}(\boldsymbol{v})$ is $\frac{\gamma}{n}$-strongly convex on $\mathcal{B}_{\sqrt{n}r}(\mathring{\boldsymbol{v}})$, where $\gamma=\frac{\theta_{\min}\sigma_{w}^{2}}{4(K^{2}+\sigma_{w}^{2})^{3/2}}$. In other words, $\widetilde{L}(\boldsymbol{v})$ is locally strongly convex in $\mathcal{B}_{\sqrt{n}r}(\mathring{\boldsymbol{v}})$. Also since $\mathcal{B}_{\sqrt{n}r}(\mathring{\boldsymbol{v}})\subset\mathcal{B}_{\sqrt{n}K}$, together with (99) we have $\min_{\boldsymbol{v}\in\mathcal{B}_{\sqrt{n}r}(\mathring{\boldsymbol{v}})}\widetilde{L}(\boldsymbol{v})\geq\min_{\boldsymbol{v}\in\mathcal{B}_{\sqrt{n}K}}\widetilde{L}(\boldsymbol{v})\geq\widetilde{L}(\mathring{\boldsymbol{v}})-2\varepsilon$. By Lemma B.1 in [50] we know if $0<2\varepsilon\leq\tfrac{(\sqrt{n}r)^{2}\gamma}{8n}$ , then $\|\tilde{\boldsymbol{v}}-\mathring{\boldsymbol{v}}\|^{2}\leq\frac{4n\varepsilon}{\gamma}\leq\frac{nr^{2}}{4}$, where $\tilde{\boldsymbol{v}}=\underset{\boldsymbol{v}\in\mathcal{B}_{\sqrt{n}K}}{\text{argmin }}\widetilde{L}(\boldsymbol{v})$. Moreover, for any $\boldsymbol{v}\in\mathcal{B}^{o}_{4\sqrt{n\varepsilon/\gamma}}(\mathring{\boldsymbol{v}})$, we have $\widetilde{L}(\boldsymbol{v})\geq\min_{\boldsymbol{v}\in\mathcal{B}_{\sqrt{n}K}}\widetilde{L}(\boldsymbol{v})+2\varepsilon$. This implies

[TABLE]

Finally, we show $L(\boldsymbol{v})$ is well-approximated by $\widetilde{L}(\boldsymbol{v})$ under event $\mathcal{A}_{1}$. Note that $L(\boldsymbol{v})-\widetilde{L}(\boldsymbol{v})=g(\Delta)$, where $g(t)=\tfrac{1}{2}[(\sqrt{x+t}-y)_{+}^{2}-(\sqrt{x}-y)_{+}^{2}]$, with $x=\frac{\|\boldsymbol{v}\|^{2}}{n}+\sigma_{w}^{2}$, $y=\frac{\boldsymbol{h}^{\top}\boldsymbol{v}}{n}$ and $\Delta=\frac{\|\boldsymbol{v}\|^{2}}{n}\left(\frac{\|\boldsymbol{g}\|^{2}}{n}-1\right)+\left(\frac{\|\boldsymbol{w}\|^{2}}{n}-\sigma_{w}^{2}\right)+2\frac{\|\boldsymbol{v}\|}{\sqrt{n}}\frac{\boldsymbol{g}^{\top}\boldsymbol{w}}{n}$. Also it is not hard to show under event $\mathcal{A}_{1}$, $|g(t)|\leq\frac{|t|}{2}\left(1+\frac{|y|}{\sigma_{w}}\right)$, $|y|\leq\sqrt{\tfrac{3}{2\delta}}K$ and $|\Delta|\leq(K^{2}+\sigma_{w}^{2}+2K)\varsigma$ for any $\boldsymbol{v}\in\mathcal{S}_{\boldsymbol{v}}(K)$. Therefore,

[TABLE]

and thus

[TABLE]

Now we are ready to turn back to $L(\boldsymbol{v})$ to show (88) and (89). Substituting (102) and (103) into (100) and (101) gives

[TABLE]

and

[TABLE]

where in (a) we use (104). For any $\varepsilon>0$, choose \varsigma\leq\min\big{\{}\tfrac{1}{2},\tfrac{\varepsilon}{6C_{K}}\big{\}} in (104) and (105). Then (88) and (89) immediately follows, since $\mathbb{P}(\mathcal{A})\to 1$. ∎

The second lemma is on the asymptotic empirical distribution of the optimal solution of auxiliary problem.

Lemma 11

Let $\psi(\cdot,\cdot)$ be a pseudo-Lipschitz function with constant $L$ and D_{\nu}:=\big{\{}\boldsymbol{v}\in\mathbb{R}^{p}:|\mathbb{E}_{\mu_{\boldsymbol{v}+\boldsymbol{\beta},\boldsymbol{\beta}}}\psi-\mathbb{E}_{\mu^{*}}\psi|\geq\nu\big{\}} as defined in (56). Then for any $K\geq\tfrac{\sigma_{*}}{\sqrt{\delta}}+\tfrac{\theta_{\min}}{4}$, $\nu>0$ and $\varepsilon\leq\frac{\gamma\nu^{2}}{192L^{2}(1+2\delta K^{2}+32(\mathbb{E}B^{2}+\sigma_{w}^{2}))\delta}$,

[TABLE]

where $\Psi_{*}$ is defined in (55).

Proof:

For any $\nu>0$, we will consider the following event:

[TABLE]

where $\mathring{\boldsymbol{v}}:=\eta(\boldsymbol{\beta}+\sigma_{*}\boldsymbol{h})-\boldsymbol{\beta}$ is the same as in (90) and $\mu^{*}$ is the joint measure of \big{(}\eta(B+\sigma_{*}H),B\big{)}, with $\eta(\cdot):=\eta(\cdot;\mu_{Y_{*}},\mu_{\sigma_{*}\Lambda/\theta_{*}})$ and $H\sim\mathcal{N}(0,1)$ independent of $B\sim\mu_{B}$.

We first show $\mathbb{P}(\mathcal{E})\to 1$, as $p\to\infty$. From (164) in the proof of Lemma 15, we have W_{2}\big{(}\mu_{\boldsymbol{h},\boldsymbol{\beta}},H\otimes B\big{)}\overset{a.s.}{\rightarrow}0, with $H\sim\mathcal{N}(0,1)$ and $B\sim\mu_{B}$. Meanwhile, (h,b)\mapsto\big{(}\eta(b+\sigma_{*}h),b\big{)} is a $\sqrt{3+2(\sigma_{*})^{2}}$-Lipschitz continuous mapping. Hence similar as (165), W_{2}\big{(}\mu_{\mathring{\boldsymbol{v}}+\boldsymbol{\beta},\boldsymbol{\beta}},\mu^{*}\big{)}\overset{a.s.}{\rightarrow}0 and by Theorem 7.12 (iv) in [56], $\mathbb{E}_{\mu_{\mathring{\boldsymbol{v}}+\boldsymbol{\beta},\boldsymbol{\beta}}}\psi\overset{a.s.}{\rightarrow}\mathbb{E}_{\mu^{*}}\psi$. Similarly, we can show

[TABLE]

Also since $\{\boldsymbol{\beta}\}_{p\in\mathbb{N}}$ is a converging sequence, $\tfrac{1}{p}\|\boldsymbol{\beta}\|^{2}\to\mathbb{E}B^{2}$. As a result, $\mathbb{P}(\mathcal{E})\to 1$ for any $\nu>0$.

Next we show conditioned on $\mathcal{E}$, it holds that

[TABLE]

where $K\geq\tfrac{\sigma_{*}}{\sqrt{\delta}}+\tfrac{\theta_{\min}}{4}$, D_{\nu}:=\big{\{}\boldsymbol{v}\in\mathbb{R}^{p}:|\mathbb{E}_{\mu_{\boldsymbol{v}+\boldsymbol{\beta},\boldsymbol{\beta}}}\psi-\mathbb{E}_{\mu^{*}}\psi|\geq\nu\big{\}} and $\varepsilon_{0}^{2}=\tfrac{\nu^{2}}{3L^{2}(1+2\delta K^{2}+32(\mathbb{E}B^{2}+\sigma_{w}^{2}))\delta}$ . Since $\psi$ is $L$ pseudo-Lipschitz, we can get

[TABLE]

where in (a) we use Cauchy-Swartz inequality and $1+\sqrt{x}+\sqrt{y}\leq\sqrt{3(1+x+y)}$. Meanwhile, conditioned on event $\mathcal{E}$, if $\boldsymbol{v}\in D_{\nu}$, then

[TABLE]

Combining (108) and (109), we know conditioned on event $\mathcal{E}$, $\tfrac{1}{n}\|\boldsymbol{v}-\mathring{\boldsymbol{v}}\|^{2}\geq\tfrac{\nu^{2}}{3L^{2}(1+2\delta K^{2}+32(\mathbb{E}B^{2}+\sigma_{w}^{2}))\delta}=\varepsilon_{0}^{2}$ for any $\boldsymbol{v}\in D_{\nu}\bigcap\mathcal{B}_{\sqrt{n}K}$.

From (107), we have

[TABLE]

On the other hand, from (89) in Lemma 10 we have

[TABLE]

if $\sqrt{n}\varepsilon_{0}\geq 8\sqrt{\frac{n\varepsilon}{\gamma}}$. Therefore, for any $\nu>0$ we can choose $\varepsilon\leq\frac{\gamma\nu^{2}}{192L^{2}(1+2\delta K^{2}+32(\mathbb{E}B^{2}+\sigma_{w}^{2}))\delta}$ and from (110) and (111) we can get (106). ∎

The last lemma in this section shows that the optimal solution of the original problem is bounded with probability converging to 1.

Lemma 12

For $K\geq\tfrac{\sigma_{*}}{\sqrt{\delta}}+\tfrac{\theta_{\min}}{4}$, we have as $p\to\infty$,

[TABLE]

Proof:

To show $\hat{\boldsymbol{v}}=\underset{\boldsymbol{v}}{\text{argmin }}C(\boldsymbol{v})$ is bounded with probability approaching 1, we use the following property: for any $a\leq K$,

[TABLE]

One can prove (113) by contradiction. If $\min_{\boldsymbol{v}\in\mathcal{B}^{o}_{\sqrt{n}a}}C(\boldsymbol{v})\leq\min_{\boldsymbol{v}\in\mathcal{B}_{\sqrt{n}K}}C(\boldsymbol{v})+\varepsilon$, it must hold that $\min_{\boldsymbol{v}\in\mathcal{B}_{\sqrt{n}K}^{C}}C(\boldsymbol{v})\leq\min_{\boldsymbol{v}\in\mathcal{B}_{\sqrt{n}K}}C(\boldsymbol{v})+\varepsilon$. Then by convexity of $C(\boldsymbol{v})$, we can always find $\boldsymbol{v}_{0}\in\mathcal{B}^{o}_{\sqrt{n}a}\bigcap\mathcal{B}_{\sqrt{n}K}$ such that $C(\boldsymbol{v}_{0})\leq\min_{\boldsymbol{v}\in\mathcal{B}_{\sqrt{n}K}}C(\boldsymbol{v})+\varepsilon$, which leads to a contradiction with $\min_{\boldsymbol{v}\in\mathcal{B}^{o}_{\sqrt{n}a}\bigcap\mathcal{B}_{\sqrt{n}K}}C(\boldsymbol{v})>\min_{\boldsymbol{v}\in\mathcal{B}_{\sqrt{n}K}}C(\boldsymbol{v})+\varepsilon$. As a result, for any $a\leq K$,

[TABLE]

where (a) follows from (113). Now we choose a\in\big{(}\tfrac{\sigma_{*}}{\sqrt{\delta}},K\big{)}. Then in the probability space of auxiliary problem (52), under event $\mathcal{A}_{2}$ [c.f. (91)] we can get $\mathcal{B}^{o}_{\sqrt{n}a}\subset\mathcal{B}^{o}_{\sqrt{n}\Delta_{a}}(\mathring{\boldsymbol{v}})$, with $\Delta_{a}=a-\tfrac{\sigma_{*}}{\sqrt{\delta}}$. Therefore, for any $\varepsilon>0$ we have

[TABLE]

where $\Psi_{*}$ is defined in (55), in (a) we use (53) and (54). Combining (114) and (115), we have for any $\varepsilon>0$,

[TABLE]

It remains to show all the three terms on the RHS of (116) converge to 0 for some $\varepsilon>0$. From (89), we know for $\varepsilon>0$ if we choose an $a$ such that $\sqrt{n}\Delta_{a}\geq 8\sqrt{\frac{n\varepsilon}{\gamma}}$, where $\gamma=\frac{\theta_{\min}\sigma_{w}^{2}}{4(K^{2}+\sigma_{w}^{2})^{3/2}}$, then

[TABLE]

Clearly, such $a$ always exists if \varepsilon\in\big{(}0,\tfrac{\gamma^{2}\theta_{\min}^{2}}{1024}\big{)} since $K\geq\tfrac{\sigma_{*}}{\sqrt{\delta}}+\tfrac{\theta_{\min}}{4}$. On the other hand, in the proof of Lemma 10 we show $\mathbb{P}(\mathcal{A}_{2}^{C})\to 0$ and from (88) we have for any $\varepsilon>0$, \mathbb{P}\Big{(}\min_{\boldsymbol{v}\in\mathcal{B}_{\sqrt{n}K}}L(\boldsymbol{v})>\Psi_{*}+\varepsilon\Big{)}\to 0. Substituting these results back to (116), we reach (112). ∎

-E Proof of Lemma 3

To obtain the probabilistic upper bound for $R_{0}^{(p)}$, we can approximate indicator function $\mathbb{I}_{x=0}$ by a series of envelope functions:

[TABLE]

where $h>0$. We can see $\psi_{h}(x)$ is an upper bound of $\mathbb{I}_{x=0}$ and satisfies $0\leq\psi_{h}(x)-\mathbb{I}_{x=0}\leq\mathbb{I}_{0<|x|<h}$, so

[TABLE]

where $\mu_{\hat{B}}$ denotes the distribution of $\eta(Y_{*})$. Moreover, $\psi_{h}(x)$ is $h^{-1}$-Lipschitz (and hence pseudo-Lipschitz by definition), so (12) can now be applied, which gives us \big{|}\mathbb{E}_{\mu_{\widehat{\boldsymbol{\beta}}}}\psi_{h}(x)-\mathbb{E}_{\mu_{\hat{B}}}\psi_{h}(x)\big{|}\overset{\mathbb{P}}{\rightarrow}0 for any fixed $h>0$. Meanwhile, by continuity of probability, we have \lim_{h\to 0}\mathbb{P}\big{(}|\eta(Y_{*})|\in(0,h)\big{)}=0. As a result, on both sides of (119) taking $p\to\infty$ and then $h\to 0$, we get for any $\varepsilon>0$,

[TABLE]

with probability approaching 1 as $p\to\infty$.

-F Proof of Lemma 4

If \mathbb{P}\big{(}\eta(Y_{*})=0\big{)}=0, then since $R_{0}^{(p)}\geq 0$, (61) trivially holds. Thus, it only remains to address the case when \mathbb{P}\big{(}\eta(Y_{*})=0\big{)}>0. Towards this end, we will utilize Lemma 5. To apply Lemma 5, we need to verify for sufficiently large $k\in[p]$, $|\hat{\boldsymbol{s}}|_{(1:k)}\prec_{S}\boldsymbol{\lambda}_{1:k}$ with probability approaching 1. For any $p,k,\ell\in\mathbb{Z}^{+}$, with $0\leq k<\ell\leq p$,

[TABLE]

where in the last step, we use (86). Here, $\mu_{\hat{S}}$ is the law of $\tau_{*}^{-1}[Y_{*}-\eta(Y_{*})]$. By condition (R.1), we know for any $\varepsilon\in(0,q_{0}^{*}]$ (recall that q_{0}^{*}=\mathbb{P}\big{(}\eta(Y_{*})=0\big{)}), there exists $\varsigma>0$ such that

[TABLE]

where to reach (a), we use $\hat{S}\overset{\text{law}}{=}\tau_{*}^{-1}[Y_{*}-\eta(Y_{*})]$ and the fact that $\eta(y)=0$ for $|y|\leq F_{|Y_{*}|}^{-1}(q_{0}^{*})$ and (b) is due to (R.1) and the fact that $t\mapsto\int_{t}^{q_{0}^{*}}F_{|Y_{*}|}^{-1}(u)du$ and $t\mapsto\int_{t}^{q_{0}^{*}}F_{\tau_{*}\Lambda}^{-1}(u)du$ are both continuous. For any fixed $\varepsilon\in(0,q_{0}^{*}]$, $\lfloor p(q_{0}^{*}-\varepsilon)\rfloor\leq\lfloor pq_{0}^{*}\rfloor-1$ for large enough $p$. Then substituting (122) into (121) we can get for large enough $p$ and any $0\leq k\leq\lfloor p(q_{0}^{*}-\varepsilon)\rfloor$,

[TABLE]

where (a) follows from (121) and (b) follows from (122). We now show the last four terms in (123) vanish as $p\to\infty$: $\int_{\lfloor pq_{0}^{*}\rfloor/p}^{q_{0}^{*}}F_{|Y_{*}|}^{-1}(u)du$ and $\int_{\lfloor pq_{0}^{*}\rfloor/p}^{q_{0}^{*}}F_{\tau_{*}\Lambda}^{-1}(u)du$ converges to 0 by DCT; $W_{2}(\mu_{\boldsymbol{s}},\mu_{S})\overset{\mathbb{P}}{\rightarrow}0$ is proved in Proposition 6; $W_{2}(\mu_{\boldsymbol{\lambda}},\mu_{\Lambda})\to 0$ since $\{\boldsymbol{\lambda}\}_{p\in\mathbb{Z}^{+}}$ is a converging sequence. As a result, for any fixed $\varepsilon\in(0,q_{0}^{*}]$, there exists $\varepsilon>0$ such that

[TABLE]

with probability approaching 1, as $p\to\infty$. Now we show conditioned on (124), there exists $k_{0}\in[\lfloor p(q_{0}^{*}-\varepsilon)\rfloor,\lfloor pq_{0}^{*}\rfloor-1]$ such that

[TABLE]

In other words, $|\hat{\boldsymbol{s}}|_{(1:k_{0}+1)}\prec_{S}\boldsymbol{\lambda}_{1:k_{0}+1}$. Such $k_{0}$ can be retrieved as follows:

1. Step 0.

Let $k_{s}$ denote the candidate for $k_{0}$ and initialize $k_{s}=\lfloor p(q_{0}^{*}-\varepsilon)\rfloor$. From (124) we know at the initial step,

[TABLE] 2. Step 1.

If $k_{s}+1=\lfloor pq_{0}^{*}\rfloor$, then we output $k_{0}=k_{s}$; otherwise we go to step 2. 3. Step 2.

If $\sum_{i=k_{s}+2}^{\lfloor pq_{0}^{*}\rfloor}|\hat{s}|_{(i)}-\sum_{i=k_{s}+2}^{\lfloor pq_{0}^{*}\rfloor}\lambda_{i}>-\frac{\varsigma}{2},$ then together with (126) we get $\max_{0\leq k\leq k_{s}}\sum_{i=k+1}^{k_{s}+1}|\hat{s}|_{(i)}-\sum_{i=k+1}^{k_{s}+1}\lambda_{i}\leq-\frac{\varsigma}{2}$. Hence, we output $k_{0}=k_{s}$; otherwise, we update $k_{s}$ and $\varsigma$ as: $k_{s}\leftarrow k_{s}+1$, $\varsigma\leftarrow\frac{\varsigma}{2}$ and return back to step 1. Clearly, (126) still holds under the updated $k_{s}$ and $\varsigma$.

Then from Lemma 5, (125) implies that $|\widehat{\boldsymbol{\beta}}|_{(1:k_{0}+1)}=0$ and thus $R_{0}^{(p)}\geq q_{0}^{*}-\varepsilon$ since $k_{0}\geq p(q_{0}^{*}-\varepsilon)-1$ by construction. Summing up, if $q_{0}^{*}>0$ then for any $\varepsilon\in(0,q_{0}^{*}]$, $R_{0}^{(p)}\geq q_{0}^{*}-\varepsilon=\mathbb{P}(\eta(Y_{*})=0)-\varepsilon$ with probability approaching 1 as $p\to\infty$. Therefore (61) is verified.

-G Proof of Lemma 5

The key is to establish the following result: for any $p$-dimensional vectors $\boldsymbol{a}$ and $\boldsymbol{\lambda}$ with $0\leq\lambda_{1}\leq\cdots\leq\lambda_{p}$, if $|\boldsymbol{a}-\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{a})|_{(1:k)}\prec_{S}\boldsymbol{\lambda}_{1:k}$ for some $k\in[p]$, then $|\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{a})|_{(1:k)}=\boldsymbol{0}$. To prove this, it suffices to show $|\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{a})|_{(k)}=0$. Assume $|\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{a})|_{(k)}\neq 0$. Define the index set $I_{k}\overset{\text{def}}{=}\{i\mid|\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{a})|_{(i)}=|\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{a})|_{(k)}\}$ and denote $\ell:=\min I_{k}$ and $m:=\max I_{k}$. According to the formula for $\partial J_{\boldsymbol{\lambda}}$ [47, Fact V.3], we have: if $|\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{a})|_{(k)}\neq 0$, then for any \boldsymbol{g}\in\partial J_{\boldsymbol{\lambda}}\big{(}\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{a})\big{)} it holds that |\boldsymbol{g}|_{(\ell:m)}$$\prec\boldsymbol{\lambda}_{\ell:m} and $\sum_{i=\ell}^{m}|\boldsymbol{g}|_{(i)}=\sum_{i=\ell}^{m}\lambda_{i}$. On the other hand, by the first order optimality condition, \boldsymbol{a}-\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{a})\in\partial J_{\boldsymbol{\lambda}}\big{(}\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{a})\big{)}. Hence, $|\boldsymbol{a}-\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{a})|_{(\ell:m)}\prec\boldsymbol{\lambda}_{\ell:m}$, i.e., for any $\ell\leq q\leq m$,

[TABLE]

and also

[TABLE]

Therefore,

[TABLE]

where the inequality follows from the condition that $|\boldsymbol{a}-\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{a})|_{(1:k)}\prec_{S}\boldsymbol{\lambda}_{1:k}$ and (127). Clearly, (129) contradicts (128). Therefore, $|\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{a})|_{(k)}=0$ and thus $|\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{a})|_{(1:k)}=\boldsymbol{0}_{k}$.

Now we are ready to prove Lemma 5. To apply the above result, we just need to express $\widehat{\boldsymbol{\beta}}$ as

[TABLE]

using the first order optimality condition of (2). Therefore, we can let $\boldsymbol{a}=\widehat{\boldsymbol{\beta}}+\hat{\boldsymbol{s}}$ and thus $\hat{\boldsymbol{s}}=\boldsymbol{a}-\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{a})$. The desired result then immediately follows.

-H Proof of Lemma 6

Similar as proof of Lemma 3, the idea is again approximating the indicator function $\mathbb{I}_{x\neq 0,y=0}$ by some Lipschitz continuous functions. Here we use:

[TABLE]

where $\psi_{h}(x)$ is defined in (118). We have

[TABLE]

Therefore, for any $h>0$,

[TABLE]

where $\mu_{\hat{B},B}$ denotes the joint distribution of $(\eta(Y_{*}),B)$ and in the last step we use (131) and $\mathbb{I}_{0<|x|<h}=\mathbb{I}_{|x|<h}-\mathbb{I}_{x=0}$. Let us compute the limit of each term on the RHS of (132). The last term converges in probability to zero due to (12). By continuity of probability, \mathbb{P}\big{(}0<|\eta(Y_{*})|<h\big{)} and $\mathbb{P}(0<|B|<h)$ converge to 0 as $h\to 0$. Following similar steps leading to (120), we can get \frac{1}{p}\sum_{i=1}^{p}\mathbb{I}_{|\hat{\beta}_{i}|<h}\overset{\mathbb{P}}{\rightarrow}\mathbb{P}\big{(}|\eta(Y_{*})|<h\big{)} and $\frac{1}{p}\sum_{i=1}^{p}\mathbb{I}_{|\beta_{i}|<h}\overset{\mathbb{P}}{\rightarrow}\mathbb{P}(|B|<h)$ if $h$ satisfies \mathbb{P}\big{(}|\eta(Y_{*})|=h\big{)}=\mathbb{P}(B=h)=0. Therefore, for such $h>0$, (132) yields the following: for any $\varepsilon>0$ there exists $p_{0}$ such that when $p>p_{0}$,

[TABLE]

where in the last step we use Assumption (A.2). Then taking $h\to 0$ along a sequence $\{h_{i}\}_{i\in\mathbb{Z}^{+}}$ with \mathbb{P}\big{(}|\eta(Y_{*})|=h_{i}\big{)}=\mathbb{P}(B=h_{i})=0 in (133), we get for any $\varepsilon>0$,

[TABLE]

with probability approaching 1 as $p\to\infty$.

-I Asymptotic Properties of $\hat{\boldsymbol{s}}$

In this section, we study the limiting properties of the following vector: $\hat{\boldsymbol{s}}=\boldsymbol{A}^{\top}(\boldsymbol{w}-\boldsymbol{A}\hat{\boldsymbol{v}}),$ where $\widehat{\boldsymbol{v}}=\text{argmin}_{\boldsymbol{v}}C(\boldsymbol{v})$. Recall that $C(\boldsymbol{v})$ is the objective function of primary problem defined in (51):

[TABLE]

and $\widehat{\boldsymbol{v}}$ is the optimal solution. The main goal is to prove Proposition 6, which characterizes the limiting empirical distribution of $(\hat{\boldsymbol{s}},\boldsymbol{\beta})$. We will follow the proof strategy in [50, Appendix E].

Proposition 6

Under the same setting as Theorem 1, define $\mu_{\hat{S},B}$ as the joint measure of $(\tau_{*}^{-1}[Y-\eta(Y;\mu_{Y_{*}},\mu_{\tau_{*}\Lambda})],B)$. It holds that $W_{2}(\mu_{\hat{\boldsymbol{s}},\boldsymbol{\beta}},\mu_{\hat{S},B})\overset{\mathbb{P}}{\rightarrow}0$.

Proof:

The first step is to obtain an alternative representation of $\hat{\boldsymbol{s}}$. Consider the event $E=\{\hat{\boldsymbol{v}}\in\mathcal{B}_{\sqrt{n}K}\}$, for some $K\geq\tfrac{\sigma_{*}}{\sqrt{\delta}}+\tfrac{\theta_{\min}}{4}$, where $\theta_{\min}$ is given in Lemma 14. It is shown in Lemma 12 that $\mathbb{P}(E)\to 1$ as $p\to\infty$. Under event $E$, we have

[TABLE]

where $C_{\boldsymbol{\lambda}}$ is defined in (187) and the last step follows from Sion’s minimax theorem [59]. By the first order optimality condition of , we know $\hat{\boldsymbol{s}}\in\partial J_{\boldsymbol{\lambda}}(\hat{\boldsymbol{v}}+\boldsymbol{\beta})$. On the other hand, it is not hard to show for any $\boldsymbol{x}\in\mathbb{R}^{p}$ and $\boldsymbol{s}\in\partial J_{\boldsymbol{\lambda}}(\boldsymbol{x})$, it holds that $\boldsymbol{s}\in C_{\boldsymbol{\lambda}}$ and $J_{\boldsymbol{\lambda}}(\boldsymbol{x})=\boldsymbol{s}^{\top}\boldsymbol{x}$. Therefore, $\hat{\boldsymbol{s}}\in C_{\boldsymbol{\lambda}}$ and $\hat{\boldsymbol{s}}^{\top}(\hat{\boldsymbol{v}}+\boldsymbol{\beta})=J_{\boldsymbol{\lambda}}(\hat{\boldsymbol{v}}+\boldsymbol{\beta})$. Then

[TABLE]

where (a) follows from first order optimality condition and the fact that $\hat{\boldsymbol{v}}\in\mathcal{B}_{\sqrt{n}K}$ under event $E$ and (b) follows from (135) and $\hat{\boldsymbol{v}}\in\min_{\boldsymbol{v}\in\mathbb{R}^{p}}C(\boldsymbol{v})$. This implies that under event $E$, which happens with probability approaching 1, $\hat{\boldsymbol{s}}\in\text{argmax}_{\boldsymbol{s}\in C_{\boldsymbol{\lambda}}}\mathcal{S}(\boldsymbol{s})$. Therefore, in order to study the limiting behavior of $\hat{\boldsymbol{s}}$, we can instead study $\text{argmax}_{\boldsymbol{s}\in C_{\boldsymbol{\lambda}}}\mathcal{S}(\boldsymbol{s})$.

The analysis of $\text{argmax}_{\boldsymbol{s}\in C_{\boldsymbol{\lambda}}}\mathcal{S}(\boldsymbol{s})$ can be carried out based on CGMT framework. First, similar as Proposition E.1 in [50], we can get for any closed set $D$ and $t\in\mathbb{R}$,

[TABLE]

and if $D$ is also convex,

[TABLE]

Here

[TABLE]

Then for any $\varepsilon,\nu>0$ and $D_{\nu}\overset{\text{def}}{=}\left\{\boldsymbol{s}:W_{2}(\mu_{\boldsymbol{s},\boldsymbol{\beta}},\mu_{\hat{S},B})\geq\nu\right\}\bigcap C_{\boldsymbol{\lambda}}$, we have

[TABLE]

where $\Psi^{*}$ is defined in (55) and step (a) follows from (136) and (137). Then combining (139) with Lemma 13, we get for any $\nu>0$, there exists $\varepsilon>0$ such that \mathbb{P}\big{(}\max_{\boldsymbol{s}\in D_{\nu}}\mathcal{S}(\boldsymbol{s})\geq\max_{\boldsymbol{s}\in C_{\boldsymbol{\lambda}}}\mathcal{S}(\boldsymbol{s})-\varepsilon\big{)}\to 0. Therefore, for any $\nu>0$ and $\varepsilon>0$,

[TABLE]

By the discussion above, the RHS of (140) converges to 0 for some $\varepsilon>0$. Since, the LHS of (140) does not depend on $\varepsilon$, this concludes the proof. ∎

Lemma 13

Under the same setting as Proposition 6, as $p\to\infty$,

[TABLE]

where $S(\boldsymbol{s})$ is defined in (138) and $\Psi^{*}$ is defined in (55). Denote $D_{\nu}:=\left\{\boldsymbol{s}:W_{2}(\mu_{\boldsymbol{s},\boldsymbol{\beta}},\mu_{\hat{S},B})\geq\nu\right\}\bigcap C_{\boldsymbol{\lambda}}$. For any $\nu>0$, there exists $\varepsilon>0$ such that

[TABLE]

Proof:

For the similar reason as introducing $\widetilde{L}(\boldsymbol{v})$ when analyzing $L(\boldsymbol{v})$ [c.f. (87) and (92) in the proof of Lemma 10], we consider the following approximation of $S(\boldsymbol{s})$:

[TABLE]

Note that $S(\boldsymbol{s})=\min_{\boldsymbol{v}\in\mathcal{B}_{\sqrt{n}K}}L(\boldsymbol{v})-\Delta(\boldsymbol{v})$ and $\widetilde{S}(\boldsymbol{s})=\min_{\boldsymbol{v}\in\mathcal{B}_{\sqrt{n}K}}\widetilde{L}(\boldsymbol{v})-\Delta(\boldsymbol{v})$, where $\Delta(\boldsymbol{v})=\tfrac{J_{\boldsymbol{\lambda}}(\boldsymbol{v}+\boldsymbol{\beta})-\boldsymbol{s}^{\top}(\boldsymbol{v}+\boldsymbol{\beta})}{n}$. Therefore, by (102)

[TABLE]

On the other hand, similar as (135) we have

[TABLE]

where $C_{\boldsymbol{\lambda}}$ is defined in (187). Using successively (144) and (145), we have for any $\varepsilon>0$,

[TABLE]

where $\delta_{p}\to 0$ as $p\to\infty$. Similarly on the other direction, we can also get for any $\varepsilon>0$, there is some $\delta_{p}^{\prime}\to 0$ such that

[TABLE]

Then combining (146) and (147) with (100), we get $\max_{\boldsymbol{s}\in C_{\boldsymbol{\lambda}}}S(\boldsymbol{s})\overset{\mathbb{P}}{\rightarrow}\Psi^{*}$ and from (144) we get $\max_{\boldsymbol{s}\in C_{\boldsymbol{\lambda}}}\widetilde{S}(\boldsymbol{s})\overset{\mathbb{P}}{\rightarrow}\Psi^{*}$.

Next, we show (142). First, the following bound holds

[TABLE]

Recall that we have already shown that the last two terms on the RHS of (148) vanish as $p\to\infty$. Therefore, it remains to show the first term also converges to 0. The main step is to establish there exist $\varsigma_{\max},\gamma>0$ such that for any $\varsigma\in(0,\varsigma_{\max})$,

[TABLE]

where

[TABLE]

and $\mathring{\boldsymbol{v}}$ is defined in (90). The convergence in (149) can be proved in exactly the same way as Theorem E.7 in [50], which deals with the case of LASSO. For simplicity, we do not re-present the proof details here. Here $C_{\boldsymbol{\lambda}}$ (the unit ball of the dual norm of $J_{\boldsymbol{\lambda}}$) plays the role of the set $\{\boldsymbol{x}\mid\|\boldsymbol{x}\|_{\infty}\leq\lambda\}$ in [50], which is the unit ball of the dual norm of $\lambda\|\cdot\|_{1}$. Now consider the event E=\{W_{2}\big{(}\mu_{\mathring{\boldsymbol{s}},\boldsymbol{\beta}},\mu_{\hat{S},B}\big{)}\leq\tfrac{\nu}{2}\}. Conditioned on $E$, it holds that for $\boldsymbol{s}\in D_{\nu}$,

[TABLE]

which indicates that $D_{\nu}\subseteq C_{\boldsymbol{\lambda}}\cap\mathcal{B}^{o}_{\frac{\sqrt{p}\nu}{2}}(\mathring{\boldsymbol{s}})$. Therefore,

[TABLE]

According to (149), the first term in (150) vanishes if $\varepsilon\leq\frac{\gamma\nu}{2}$. For the second term, it is not hard to show (H,B)\mapsto\big{(}\tau_{*}^{-1}[(B+\sigma_{*}H)-\eta(B+\sigma_{*}H;\mu_{Y_{*}},\mu_{\tau_{*}\Lambda})],B\big{)} is a Lipschitz continuous mapping and from (164), W_{2}\big{(}\mu_{\boldsymbol{h},\boldsymbol{\beta}},H\otimes B\big{)}\overset{\mathbb{P}}{\rightarrow}0, so similar as (165) we can get W_{2}\big{(}\mu_{\mathring{\boldsymbol{s}},\boldsymbol{\beta}},\mu_{\hat{S},B}\big{)}\overset{\mathbb{P}}{\rightarrow}0 and thus $\mathbb{P}(E^{C})\to 0$. Therefore, from (150), for any $\nu>0$, there exists $\varepsilon_{0}>0$ such that any $\varepsilon\leq\varepsilon_{0}$ satisfies \mathbb{P}\big{(}\max_{\boldsymbol{s}\in D_{\nu}}\widetilde{S}(\boldsymbol{s})\geq\max_{\boldsymbol{s}\in C_{\boldsymbol{\lambda}}}\widetilde{S}(\boldsymbol{s})-\varepsilon\big{)}. Substituting this back to (148), we finish the proof. ∎

-J Properties of Limiting Scalar Problem

It turns out that the limiting behavior of (2) is fully captured by (55). In this section, we study the key properties of (55).

Lemma 14

The minimax problem (55) has a unique optimal solution $(\sigma_{*},\theta_{*})$, which is also the unique solution to the equation:

[TABLE]

where $Y=B+\sigma H$, with $H\sim\mathcal{N}(0,1)$ independent of $B\sim\mu_{B}$. Besides, there exists $\theta_{\min}>0$ such that $\theta^{*}\geq\theta_{\min}$.

Proof:

The proof includes two steps: (I) show the saddle point of $\Psi(\sigma,\theta)$ exists and is unique and it is also the unique optimal solution of the minimax problem (55), (II) show $(\sigma_{*},\theta_{*})$ is the saddle point of $\Psi(\sigma,\theta)$ if and only if it is the solution to (151).

We first show the set of saddle points of $\Psi(\sigma,\theta)$ is nonempty and compact, using Proposition 5.5.7 of [60]. To apply this result, it suffices to check: (i) $\Psi(\cdot,\theta)$ and $-\Psi(\sigma,\cdot)$ are convex and closed for any fixed $\theta\geq 0$ and $\sigma\geq\sigma_{w}$, (ii) there exists some $\overline{\theta}\geq 0,\,\overline{\sigma}\geq\sigma_{w}$ and $\gamma_{1},\gamma_{2}\in\mathbb{R}$ such that the level sets $\{\sigma\geq\sigma_{w}\mid\Psi(\sigma,\overline{\theta})\leq\gamma_{1}\}$ and $\{\theta\geq 0\mid-\Psi(\overline{\sigma},\theta)\leq\gamma_{2}\}$ are both non-empty and compact. From Lemma 16, $\mathcal{F}(\sigma,\theta)$ is convex-concave and continuously differentiable with respect to $\sigma$ and $\theta$. Therefore, condition (i) is satisfied. Also partial derivatives of $\mathcal{F}(\sigma,\theta)$ can be computed as

[TABLE]

using (55), (178) and (177). Next we show $\{\sigma\geq\sigma_{w}\mid\Psi(\sigma,\overline{\theta})\leq\gamma_{1}\}$ is non-empty and compact for some $\overline{\theta}\geq 0$ and $\gamma_{1}\in\mathbb{R}$. First, we have

[TABLE]

where (a) follows from the identity $\text{Prox}_{\tau\boldsymbol{\lambda}}(\boldsymbol{x})=\tau\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{x}/\tau)$, (b) follows from the non-expansiveness of proximal operator and (c) is a consequence of (188), where $\bar{\lambda}=\tfrac{1}{p}\sum_{i=1}^{p}\lambda_{i}$. Plugging the bound (154) into (152) gives

[TABLE]

where $\Phi(z)$ is CDF of standard Gaussian. When $\mathbb{E}\Lambda>0$ , from (155) we know there exists $\theta_{1}>0$ and $\sigma_{1}\geq\sigma_{w}$ such that $\frac{\partial\Psi(\sigma,\theta_{1})}{\partial\sigma}\geq\frac{\theta_{1}}{8}$ for all $\sigma\geq\sigma_{1}$; when $\mathbb{E}\Lambda=0$, by our assumption we must have $\delta>1$, so from (157) we have $\frac{\partial\Psi(\sigma,\theta)}{\partial\sigma}=\frac{\theta}{2\sigma^{2}}[(1-\tfrac{1}{\delta})\sigma^{2}-\sigma_{w}^{2}]$ for any $\theta>0$, $\sigma\geq\sigma_{w}$ implying $\frac{\partial\Psi(\sigma,\theta)}{\partial\sigma}\geq\tfrac{\theta}{4}(1-\tfrac{1}{\delta})$ for all $\sigma\geq\sqrt{\frac{2\delta}{\delta-1}}\sigma_{w}$. Therefore, there exists $\overline{\theta}>0$, $c>0$ and $K\geq\sigma_{w}$ such that $\frac{\partial\Psi(\sigma,\overline{\theta})}{\partial\sigma}\geq c$ for any $\sigma\geq K$. This means that $\Psi(\sigma,\overline{\theta})>\Psi(K,\overline{\theta})$ for all $\sigma>K$, so the set $\{\sigma\geq\sigma_{w}\mid\Psi(\sigma,\overline{\theta})\leq\Psi(K,\overline{\theta})\}\subset[\sigma_{w},K]$ and it is non-empty (include at least one point $\sigma=K$) and closed since $\Psi(\cdot,\overline{\theta})$ is a closed function. As a result, we can take $\gamma_{1}=\Psi(K,\overline{\theta})$ and the level set $\{\sigma\geq\sigma_{w}\mid\Psi(\sigma,\overline{\theta})\leq\gamma_{1}\}$ is non-empty and compact. On the other hand, we can show $\{\theta\geq 0\mid-\Psi(\sigma_{w},\theta)\leq 0\}$ is non-empty and compact. First since $\Psi(\sigma_{w},\cdot)$ is 1-strongly concave and continuously differentiable, we have for any $\theta\geq 0$,

[TABLE]

where in the last step we use $\Psi(\sigma_{w},0)=0$ and $\frac{\partial\Psi(\sigma,0)}{\partial\theta}=\sigma_{w}+\frac{\mathbb{E}B^{2}}{2\sigma_{w}\delta}$, which can be deduced from (55) and (153). Then the level set \{\theta\geq 0\mid-\Psi(\sigma_{w},\theta)\leq 0\}\subset\Big{[}0,2\big{(}\sigma_{w}+\tfrac{\mathbb{E}B^{2}}{2\sigma_{w}\delta}\big{)}\Big{]} and it is non-empty (include at least one point $\theta=0$) and closed since $\Psi(\sigma_{w},\cdot)$ is a closed function. Letting $\gamma_{2}=0$, we verify condition (ii).

Up to now, we have proved the existence and boundedness of saddle points of $\Psi(\sigma,\theta)$. Next we prove the uniqueness. To do this, it suffices to show the optimal solution of $\min_{\sigma\geq\sigma_{w}}\max_{\theta\geq 0}\Psi(\sigma,\theta)$ is bounded and unique, then the uniqueness of saddle points follows due to the fact that each saddle point of $\Psi(\sigma,\theta)$ is also an optimal solution of $\min_{\sigma\geq\sigma_{w}}\max_{\theta\geq 0}\Psi(\sigma,\theta)$ [60, Proposition 3.4.1]. First, we show that any $\sigma_{*}$ should be bounded. Indeed, from the verification of condition (ii) above, we know there exists $c>0$ and $K\geq\sigma_{w}$ such that for any $\sigma\geq K$,

[TABLE]

so we must have

[TABLE]

otherwise, $\min_{\sigma\geq\sigma_{w}}\max_{\theta\geq 0}\Psi(\sigma,\theta)>\min_{\sigma\in[\sigma_{w},K]}\max_{\theta\geq 0}\Psi(\sigma,\theta)$ leading to a contradiction. On the other hand, we can also show $\theta_{*}(\sigma):=\underset{\theta\geq 0}{\text{argmax }}\Psi(\sigma,\theta)$ is uniformly bounded for $\sigma\in[\sigma_{w},C_{1}]$. To see this, note from (153)

[TABLE]

As a result, $\max_{\theta\geq 0}\Psi(\sigma,\theta)=\max_{\theta\in[0,C_{2}]}\Psi(\sigma,\theta)$ for $\sigma\in[\sigma_{w},C_{1}]$. Therefore, by Berge Maximum Theorem [61, Theorem 17.31], $\theta_{*}(\sigma)$ is an upper hemicontinuous correspondence on $[\sigma_{w},C_{1}]$. By strong concavity of $\Psi(\sigma,\cdot)$, $\sigma\mapsto\theta_{*}(\sigma)$ is a function (i.e., single-valued correspondence). As a result, one can easily check by definition that $\theta_{*}(\sigma)$ is continuous on $[\sigma_{w},C_{1}]$. Besides, we can get $\theta_{*}(\sigma)>0$ for any $\sigma\in[\sigma_{w},C_{1}]$. Indeed from (153) when $\mathbb{P}(\Lambda=0)<1$, \frac{\partial\Psi(\sigma,0)}{\partial\theta}=\frac{1}{2}\Big{(}\frac{\sigma_{w}^{2}}{\sigma}+\sigma\Big{)}+\frac{\mathbb{E}B^{2}}{2\sigma\delta}>0 and when $\mathbb{P}(\Lambda=0)=1$, $\delta\geq 1$, $\frac{\partial\Psi(\sigma,0)}{\partial\theta}\geq\tfrac{\sigma_{w}^{2}}{2\sigma}+\tfrac{\sigma}{2}(1-\tfrac{1}{\delta})>0$. Therefore, there exists $\theta_{\min}>0$ such that

[TABLE]

for any $\sigma\in[\sigma_{w},C_{1}]$. Since $\theta^{*}(\sigma)\in[\theta_{\min},C_{2}]$ for any $\sigma\in[\sigma_{w},C_{1}]$, we get $\max_{\theta\geq 0}\Psi(\sigma,\theta)=\max_{\theta\in[\theta_{\min},C_{2}]}\Psi(\sigma,\theta)$ for any $\sigma\in[\sigma_{w},C_{1}]$. On the other hand, $\Psi(\cdot,\theta)$ is $\tfrac{\sigma_{w}^{2}\theta_{\min}}{C_{1}^{3}}$-strongly convex on $[\sigma_{w},C_{1}]$ for any fixed $\theta\in[\theta_{\min},C_{2}]$, so we can check by definition that the function $\max_{\theta\in[\theta_{\min},C_{2}]}\Psi(\cdot,\theta)$ is also $\tfrac{\sigma_{w}^{2}\theta_{\min}}{C_{1}^{3}}$-strongly convex on $[\sigma_{w},C_{1}]$. We conclude that $\sigma\mapsto\max_{\theta\geq 0}\Psi(\sigma,\theta)$ is $\tfrac{\sigma_{w}^{2}\theta_{\min}}{C_{1}^{3}}$-strongly convex on $[\sigma_{w},C_{1}]$, since $\max_{\theta\geq 0}\Psi(\sigma,\theta)=\max_{\theta\in[\theta_{\min},C_{2}]}\Psi(\sigma,\theta)$. Recall that any optimal solution $\sigma_{*}=\underset{\sigma\geq\sigma_{w}}{\text{argmin }}\max_{\theta\geq 0}\Psi(\sigma,\theta)$ should lie in $[\sigma_{w},C_{1}]$, so the uniqueness holds.

Finally, we show $(\sigma_{*},\theta_{*})$ is a saddle point of $\Psi(\sigma,\theta)$ if and only if it is a solution of (151). From (152) and (153), $\frac{\partial\Psi(\sigma_{w},\theta)}{\partial\sigma}\leq 0$ for any $\theta\geq 0$ and $\frac{\partial\Psi(\sigma,0)}{\partial\theta}\geq 0$ for any $\sigma\geq\sigma_{w}$. Since $\Psi(\sigma,\theta)$ is convex-concave and continuously differentiable, by first order optimality condition we know $(\sigma_{*},\theta_{*})$ is a saddle point if and only if $\frac{\partial\Psi(\sigma_{*},\theta_{*})}{\partial\sigma}=\frac{\partial\Psi(\sigma_{*},\theta_{*})}{\partial\theta}=0$. On the other hand, from (55), (167) and (168) we can get

[TABLE]

Note that (157) and (158) are actually the scalar representation of (152) and (153). Setting the RHS of (157) and (158) to be zero, we can get (151). ∎

-K Moreau Envelope of $J_{\boldsymbol{\lambda}}$

Recall that for $\tau>0$, the* *Moreau envelope of $J_{\boldsymbol{\lambda}}(\boldsymbol{x})$ is:

[TABLE]

and the corresponding optimal solution is the proximal operator $\text{Prox}_{\tau\boldsymbol{\lambda}}(\boldsymbol{y})$. In this section, we study the limiting behavior of the following function:

[TABLE]

where $(\sigma,\theta)\in\mathbb{R}_{>0}\times\mathbb{R}_{\geq 0}$, $\boldsymbol{h}\sim\mathcal{N}(\boldsymbol{0},\boldsymbol{I}_{p})$ and $\boldsymbol{\beta}$ and $\boldsymbol{\lambda}$ are the same as in (1) and (2).

Lemma 15

Consider a sequence of instances $\{\boldsymbol{h}^{(p)},\boldsymbol{\beta}^{(p)},\boldsymbol{\lambda}^{(p)}\}_{p\in\mathbb{Z}^{+}}$, where $\boldsymbol{h}^{(p)}\sim\mathcal{N}(\boldsymbol{0},\boldsymbol{I}_{p})$ are all independent and $\{\boldsymbol{\beta}^{(p)}\}_{p\in\mathbb{Z}^{+}}$, $\{\boldsymbol{\lambda}^{(p)}\}_{p\in\mathbb{Z}^{+}}$ are both converging sequences with limiting measure $\mu_{B}$ and $\mu_{\Lambda}$. As $p\to\infty,$ for every $(\sigma,\theta)\in\mathbb{R}_{>0}\times\mathbb{R}_{\geq 0}$,

[TABLE]

and

[TABLE]

where

[TABLE]

Here, $Y=B+\sigma H$, with $H\sim\mathcal{N}(0,1)$, $B\sim\mu_{B}$ independent and $\eta(\cdot)$ is the limiting scalar function in Proposition 1.

Proof:

For notational simplicity, we will omit the superscript “$(p)$” in $\boldsymbol{h}^{(p)},\boldsymbol{\beta}^{(p)}$ and $\boldsymbol{\lambda}^{(p)}$. We first show the empirical measure $\mu_{\boldsymbol{h},\boldsymbol{\beta}}$ converges almost surely to $H\otimes B$ under Wasserstein-2 distance. Let $g(x,y)$ be a bounded and continuous test function. By strong law of large number for triangular array [58, Theorem 2.1], we can get $\frac{1}{p}\sum_{i=1}^{p}[g(h_{i},\beta_{i})-\mathbb{E}_{H}g(H,\beta_{i})]\overset{a.s.}{\rightarrow}0$. For any $y$, $\mathbb{E}_{H}g(H,y)$ is bounded and it is not hard to show $y\mapsto\mathbb{E}_{H}g(H,y)$ is continuous: we know $g(h,y)$ is uniformly continuous over any compact set in $\mathbb{R}^{2}$, so for C=\Phi^{-1}\big{(}1-\frac{\varepsilon}{8\|g\|_{\infty}}\big{)} and any $y_{0}\in\mathbb{R}$, $\varepsilon>0$, there exists $\delta>0$ such that $|g(h,y)-g(h,y_{0})|\leq\frac{\varepsilon}{2}$ whenever $|y-y_{0}|\leq\delta$ and $|h|\leq C$. Hence,

[TABLE]

This shows the continuity of $y\mapsto\mathbb{E}_{H}[g(H,y)]$. Therefore, $\frac{1}{p}\sum_{i=1}^{p}\mathbb{E}_{H}[g(H,\beta_{i})]\to\mathbb{E}_{H,B}[g(H,B)]$ and we get for any bounded and continuous $g(x,y)$, $\frac{1}{p}\sum_{i=1}^{p}g(h_{i},\beta_{i})\overset{a.s.}{\rightarrow}\mathbb{E}_{H,B}[g(H,B)]$ indicating the almost sure weak convergence of $\mu_{\boldsymbol{h},\boldsymbol{\beta}}$ to $H\otimes B$. On the other hand, by strong law of large number again, $\frac{1}{p}\sum_{i=1}^{p}h_{i}^{2}+\beta_{i}^{2}\overset{a.s.}{\rightarrow}1+\mathbb{E}B^{2}$. Therefore, by Theorem 7.12 (iii) in [56],

[TABLE]

Then we can show W_{2}\big{(}\mu_{\boldsymbol{y}},\mu_{Y}\big{)}\overset{a.s.}{\rightarrow}0, where $\boldsymbol{y}:=\boldsymbol{\beta}+\sigma\boldsymbol{h}$. Indeed,

[TABLE]

and since W_{2}\big{(}\mu_{\boldsymbol{h},\boldsymbol{\beta}},H\otimes B\big{)}\overset{a.s.}{\rightarrow}0, we get W_{2}\big{(}\mu_{\boldsymbol{y}},\mu_{Y}\big{)}\overset{a.s.}{\rightarrow}0. Similarly, for $\theta>0$ we can show W_{2}\big{(}\mu_{\sigma\boldsymbol{\lambda}/\theta},\mu_{\sigma\Lambda/\theta}\big{)}\to 0 from our assumption that W_{2}\big{(}\mu_{\boldsymbol{\lambda}},\mu_{\Lambda}\big{)}\to 0.

Now we can prove (161). For $\theta=0$, it directly follows from (160) and (163) that $\mathcal{F}_{p}(\sigma,\theta)=\mathcal{F}(\sigma,\theta)=0$. For $\theta>0$, we have W_{2}\big{(}\mu_{\boldsymbol{y}},\mu_{Y}\big{)}\overset{a.s.}{\rightarrow}0 and W_{2}\big{(}\mu_{\sigma\boldsymbol{\lambda}/\theta},\mu_{\sigma\Lambda/\theta}\big{)}\to 0. Then from Proposition 1, we can get (161) by letting $\tau=\sigma/\theta$.

To prove (162), first observe that:

[TABLE]

By strong law of large number (SLLN), we have

[TABLE]

In other words,

[TABLE]

Then by (161) and generalized dominated convergence theorem (GDCT) (Theorem 19 in Sec. 4.4 of [57]), we have

[TABLE]

which is exactly (162). ∎

Lemma 16

$\mathcal{F}(\sigma,\theta)$* is convex-concave and continuously differentiable with respect to both $\sigma$ and $\theta$ on $\mathbb{R}_{>0}\times\mathbb{R}_{\geq 0}$, with partial derivatives:*

[TABLE]

where $B\sim\mu_{B}$, $\Lambda\sim\mu_{\Lambda}$, $Y=B+\sigma H$, with $H\sim\mathcal{N}(0,1)$ independent of $B$. Here, when $\theta>0$, $\eta(\cdot)=\eta(\cdot;F_{Y},F_{\sigma\Lambda/\theta})$; when $\theta=0$, we let $\eta(y)=0$, if $\mathbb{P}(\Lambda=0)<1$ and $\eta(y)=y$, if $\mathbb{P}(\Lambda=0)=1$.

Proof:

When $\mathbb{P}(\Lambda=0)=1$, from (163) we have $\mathcal{F}(\sigma,\theta)=0$ for any $(\sigma,\theta)\in\mathbb{R}_{>0}\times\mathbb{R}_{\geq 0}$, so $\frac{\partial\mathcal{F}(\sigma,\theta)}{\partial\sigma}=\frac{\partial\mathcal{F}(\sigma,\theta)}{\partial\theta}=0$. In this case, all the results trivially hold. Therefore, it suffices to consider $\mathbb{P}(\Lambda=0)<1$.

We first prove the convexity of $\mathcal{F}(\sigma,\theta)$. Note $\mathcal{F}_{p}(\sigma,\theta)$ [defined in (160)] can be rewritten as:

[TABLE]

Since $h(\boldsymbol{v})=\frac{\theta}{2}\|\boldsymbol{v}-\boldsymbol{h}\|^{2}$ is a convex function (due to $\theta\geq 0$), $(\sigma,\boldsymbol{v})\mapsto\frac{\sigma\theta}{2}\|\boldsymbol{v}/\sigma-\boldsymbol{h}\|^{2}$ is convex since it is the perspective function of $h(\boldsymbol{v})$. Therefore, the objective function in (169) is jointly convex w.r.t. $(\sigma,\boldsymbol{v})$ and after partial minimization over $\boldsymbol{v}$, $\mathcal{F}_{p}(\cdot,\theta)$ is still convex. On the other hand, $\mathcal{F}_{p}(\sigma,\theta)$ as a function of $\theta$ is the infimum of a family of linear functions of $\theta$, so $\mathcal{F}_{p}(\sigma,\cdot)$ is concave. Denote $\overline{\mathcal{F}}_{p}(\sigma,\theta):=\mathbb{E}_{\boldsymbol{h}}\mathcal{F}_{p}(\sigma,\theta)$. Clearly, $\overline{\mathcal{F}}_{p}(\sigma,\theta)$ and $F(\sigma,\theta)$ are still convex-cancave, since taking expectation and limit preserves convexity.

Then we show for fixed $\sigma\in\mathbb{R}_{>0}$, $\mathcal{F}_{p}(\sigma,\cdot)$ is continuously differentiable on $\mathbb{R}_{\geq 0}$. We follow the same argument as Theorem 2.26 in [62]. Denote $\boldsymbol{y}:=\boldsymbol{\beta}+\sigma\boldsymbol{h}$ and

[TABLE]

where $\varepsilon\geq-\theta$. On one hand, we have

[TABLE]

On the other hand,

[TABLE]

From Lemma 17 there exists $C>0$ such that $\frac{1}{p}\left|\frac{\partial\|\text{Prox}_{\sigma\boldsymbol{\lambda}/\theta}(\boldsymbol{y})\|^{2}}{\partial\theta}\right|\leq C$ for any $\theta\geq 0$. Combined with (171) and (172), this indicates $-\frac{C\varepsilon^{2}}{2\sigma}\leq g(\varepsilon)\leq 0.$ Substituting the boundedness of $g(\varepsilon)$ into (170) yields for any $\theta\geq 0$,

[TABLE]

Also $\frac{\partial\mathcal{F}_{p}(\sigma,\theta)}{\partial\theta}$ is continuous, which follows from (173), (183) and (184).

Following the same procedure as above, we can also show for fixed $\theta\in\mathbb{R}_{\geq 0}$, $\mathcal{F}_{p}(\cdot,\theta)$ is continuously differentiable on $\mathbb{R}_{>0}$ with

[TABLE]

From (173) and (174), we can get $\left|\frac{\partial\mathcal{F}_{p}(\sigma,\theta)}{\partial\theta}\right|\leq\frac{4(\sigma^{2}\|\boldsymbol{h}\|^{2}+\|\boldsymbol{\beta}\|^{2})}{\sigma p}$ and $\left|\frac{\partial\mathcal{F}_{p}(\sigma,\theta)}{\partial\sigma}\right|\leq\frac{3\theta\|\boldsymbol{\beta}\|^{2}+\sigma^{2}(2+\theta)\|\boldsymbol{h}\|^{2}}{\sigma^{2}p}.$ Since both bounds have finite expectation, by dominated convergence theorem (DCT) we have:

[TABLE]

and

[TABLE]

We now show $\mathcal{F}(\sigma,\cdot)$ and $\mathcal{F}(\cdot,\theta)$ are continuously differentiable on $\mathbb{R}_{\geq 0}$ and $\mathbb{R}_{>0}$, respectively. In particular, we only present the proof for $\mathcal{F}(\sigma,\cdot)$ and the case of $\mathcal{F}(\cdot,\theta)$ can be derived following same approach. The key is to establish the uniform convergence of $\frac{\partial\overline{\mathcal{F}}_{p}(\sigma,\theta)}{\partial\theta}$ to $\frac{\partial\overline{\mathcal{F}}_{\infty}(\sigma,\theta)}{\partial\theta}$ on $\theta\geq 0$, where $\frac{\partial\overline{\mathcal{F}}_{\infty}(\sigma,\theta)}{\partial\theta}:=\lim_{p\to\infty}\tfrac{\partial\overline{\mathcal{F}}_{p}(\sigma,\theta)}{\partial\theta}$.

First consider the case $\theta>0$. Let $[a,b]\subset\mathbb{R}_{>0}$ and $\theta_{1},\theta_{2}\in[a,b]$. We have

[TABLE]

where (a) follows from (173) and intermediate value theorem, with $\theta^{\prime}\in[\theta_{1},\theta_{2}]$, (b) follows from (183) and (184), and in (c), $C>0$ is some constant that only depends on $\sigma$, $a$ and $b$. Therefore, $\tfrac{\partial\overline{\mathcal{F}}_{p}(\sigma,\theta)}{\partial\theta}$ is $C$-Lipschitz continuous on $\theta\in[a,b]$ for any $p\in\mathbb{Z}^{+}$. Meanwhile, we can show \big{\{}\tfrac{\partial\overline{\mathcal{F}}_{p}(\sigma,\theta)}{\partial\theta}\big{\}}_{p\in\mathbb{Z}^{+}} converges pointwise to $\frac{\partial\overline{\mathcal{F}}_{\infty}(\sigma,\theta)}{\partial\theta}$ on $\theta\geq 0$ following the proof of (166). Then for any $\varepsilon>0$ we can construct a $\tfrac{\varepsilon}{3C}$-epsilon net $\mathcal{E}$ such that for any $\theta\in[a,b]$, there exists $z\in\mathcal{E}$ satisfying $|\theta-z|\leq\tfrac{\varepsilon}{3C}$. Clearly, the cardinality $|\mathcal{E}|<\infty$. Therefore, for any $\varepsilon>0$ there exists $p_{0}$ such that for any $m,p\geq p_{0}$, $\max_{z\in\mathcal{E}}\left|\tfrac{\partial\overline{\mathcal{F}}_{m}(\sigma,z)}{\partial\theta}-\tfrac{\partial\overline{\mathcal{F}}_{p}(\sigma,z)}{\partial\theta}\right|\leq\frac{\varepsilon}{3}$. Hence for any $\varepsilon>0$, $\theta\in[a,b]$ and $m,p\geq p_{0}$,

[TABLE]

which implies the uniform convergence of $\frac{\partial\overline{\mathcal{F}}_{p}(\sigma,\theta)}{\partial\theta}$ when $\theta\in[a,b]$. Meanwhile, by DCT and continuity of $\frac{\partial\mathcal{F}_{p}(\sigma,\theta)}{\partial\theta}$ for any fixed $\boldsymbol{h}$, $\frac{\partial\overline{\mathcal{F}}_{p}(\sigma,\theta)}{\partial\theta}$ is also continuous w.r.t. $\theta$. Then we can apply Theorem 7.12 in [63] to obtain $\frac{\partial\overline{\mathcal{F}}_{\infty}(\sigma,\theta)}{\partial\theta}$ is continuous on $[a,b]$. Since $[a,b]$ above can be arbitrary subset of $\mathbb{R}_{>0},$ we actually obtain that $\frac{\partial\overline{\mathcal{F}}_{\infty}(\sigma,\theta)}{\partial\theta}$ is continuous when $\theta>0$.

Next we handle the case when $\theta=0$. From (175), (183) and (184), we can get for any $\theta\geq 0$

[TABLE]

where in the first equality we use DCT and the last inequality follows from Fact 1. Hence, $\frac{\partial\overline{\mathcal{F}}_{p}(\sigma,\theta)}{\partial\theta}$ is non-increasing on $\theta\geq 0$, with $\frac{\partial\overline{\mathcal{F}}_{p}(\sigma,0)}{\partial\theta}=\mathbb{E}\frac{\|\boldsymbol{y}\|^{2}}{2\sigma p}$ and $\lim_{\theta\to\infty}\frac{\partial\overline{\mathcal{F}}_{p}(\sigma,\theta)}{\partial\theta}=0$. Since $\frac{\partial\overline{\mathcal{F}}_{p}(\sigma,\theta)}{\partial\theta}$ converges to $\frac{\partial\overline{\mathcal{F}}_{\infty}(\sigma,\theta)}{\partial\theta}$ pointwise, $\frac{\partial\overline{\mathcal{F}}_{\infty}(\sigma,\theta)}{\partial\theta}$ is also non-increasing with $\frac{\partial\overline{\mathcal{F}}_{\infty}(\sigma,0)}{\partial\theta}=\frac{\mathbb{E}B^{2}+\sigma^{2}}{2\sigma}$ and $\lim_{\theta\to\infty}\frac{\partial\overline{\mathcal{F}}_{\infty}(\sigma,\theta)}{\partial\theta}=0$. On the other hand, from (175)

[TABLE]

where step (a) follows from (188), with $\overline{\boldsymbol{\lambda}}=\frac{\sum_{i=1}^{p}\lambda_{i}}{p}\boldsymbol{1}_{p}$. Hence taking $p\to\infty$ on both sides above, we have

[TABLE]

Then taking $\theta\to 0^{+}$, we get

[TABLE]

where in step (a) we use DCT and $\mathbb{E}\Lambda>0$. Since $\frac{\partial\overline{\mathcal{F}}_{\infty}(\sigma,\theta)}{\partial\theta}$ is non-increasing on $\theta\geq 0$, we can get $\lim_{\theta\to 0^{+}}\frac{\partial\overline{\mathcal{F}}_{\infty}(\sigma,\theta)}{\partial\theta}=\frac{\partial\overline{\mathcal{F}}_{\infty}(\sigma,0)}{\partial\theta}.$ Recall that we have already proved $\frac{\partial\overline{\mathcal{F}}_{\infty}(\sigma,\theta)}{\partial\theta}$ is continuous when $\theta>0$, so $\frac{\partial\overline{\mathcal{F}}_{\infty}(\sigma,\theta)}{\partial\theta}$ is continuous on $\mathbb{R}_{\geq 0}$.

Up to now, we have shown \Big{\{}\frac{\partial\overline{\mathcal{F}}_{p}(\sigma,\theta)}{\partial\theta}\Big{\}}_{p\in\mathbb{Z}^{+}}, $\frac{\partial\overline{\mathcal{F}}_{\infty}(\sigma,\theta)}{\partial\theta}$ are all bounded, non-increasing and continuous functions on $\mathbb{R}_{\geq 0}$ and also \Big{\{}\frac{\partial\overline{\mathcal{F}}_{p}(\sigma,\theta)}{\partial\theta}\Big{\}}_{p\in\mathbb{Z}^{+}} converges to $\frac{\partial\overline{\mathcal{F}}_{\infty}(\sigma,\theta)}{\partial\theta}$ pointwise on $\theta\geq 0$. Therefore, following the proof of Glivenko-Cantelli theorem (for a reference, see Theorem 19.1 in [53]) we can show $\frac{\partial\overline{\mathcal{F}}_{p}(\sigma,\theta)}{\partial\theta}\to\frac{\partial\overline{\mathcal{F}}_{\infty}(\sigma,\theta)}{\partial\theta}$ converges uniformly on $\theta\geq 0$ . In (162) we prove the pointwise convergence $\overline{\mathcal{F}}_{p}(\sigma,\theta)\to\mathcal{F}(\sigma,\theta)$. Then using Theorem 7.17 and 7.12 in [63] together with (175), we get $\mathcal{F}(\sigma,\cdot)$ is continuously differentiable on $\mathbb{R}_{\geq 0}$, with

[TABLE]

where $\boldsymbol{y}=\boldsymbol{\beta}+\sigma\boldsymbol{h}$. Repeating the same procedure, we can also prove that $\mathcal{F}(\cdot,\theta)$ is continuously differentiable on $\mathbb{R}_{>0}$, with

[TABLE]

Finally, we compute the limit in (177) and (157). From Proposition 1 and (165), we have $\tfrac{\|\text{Prox}_{\sigma\boldsymbol{\lambda}/\theta}(\boldsymbol{y})-\eta(\boldsymbol{y})\|_{2}^{2}}{p}\overset{a.s.}{\rightarrow}0$ as $p\to\infty$, where $\eta:=\eta(\cdot;\mu_{B+\sigma H},\mu_{\sigma\Lambda/\tau})$. In addition,

[TABLE]

so we can get

[TABLE]

where the last step follows from Theorem 7.12 (iv) in [56] after combining (164) with the fact that $[\eta(b+\sigma h)-b]^{2}\leq C(1+h^{2}+b^{2})$ for any $h,b\in\mathbb{R}$ and some $C>0$. Then similar as (166), we can obtain

[TABLE]

On the other hand, we can also get

[TABLE]

and

[TABLE]

where in the last step we use Stein’s lemma. Substituting (180) and (182) into (177) and (178), we reach at (167) and (168). ∎

Lemma 17

For any $\theta\geq 0$, $\boldsymbol{y}\in\mathbb{R}^{p}$ and $\boldsymbol{\lambda}\in\mathbb{R}_{\geq 0}^{p}$, we have

[TABLE]

and

[TABLE]

Here when $\boldsymbol{\lambda}\neq\boldsymbol{0}$ and $\theta=0$, we let $\text{Prox}_{\sigma\boldsymbol{\lambda}/\theta}(\boldsymbol{y}):=\boldsymbol{0}$.

Proof:

When $\boldsymbol{\lambda}=\boldsymbol{0}$, we have $\text{Prox}_{\sigma\boldsymbol{\lambda}/\theta}(\boldsymbol{y})=\boldsymbol{y}$, so $\|\text{Prox}_{\sigma\boldsymbol{\lambda}/\theta}(\boldsymbol{y})\|^{2}=\boldsymbol{y}^{\top}\text{Prox}_{\sigma\boldsymbol{\lambda}/\theta}(\boldsymbol{y})=\|\boldsymbol{y}\|^{2}$ and $\tfrac{\partial\|\text{Prox}_{\sigma\boldsymbol{\lambda}/\theta}(\boldsymbol{y})\|^{2}}{\partial\theta}=\tfrac{\partial\boldsymbol{y}^{\top}\text{Prox}_{\sigma\boldsymbol{\lambda}/\theta}(\boldsymbol{y})}{\partial\theta}=0$.

Next we consider $\boldsymbol{\lambda}\neq\boldsymbol{0}$. From Lemma 2.3 and 2.4 of [9], for any $\boldsymbol{a}\in\mathbb{R}^{p}$ satisfying $0\leq a_{1}\leq a_{2}\leq\cdots\leq a_{p}$,

[TABLE]

where I_{j}\overset{\text{def}}{=}\big{\{}k\in[p]\mid|[\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{a})]_{k}|=|[\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{a})]_{j}|\text{ and }[\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{a})]_{k}\neq 0\big{\}}. Therefore,

[TABLE]

and

[TABLE]

On the other hand, by Fact 1, $\|\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{a})\|^{2}$ and $\boldsymbol{a}^{\top}\text{Prox}_{\boldsymbol{\lambda}}(\boldsymbol{a})$ only depend on $\mu_{\boldsymbol{\lambda}}$ and $\mu_{|\boldsymbol{a}|}$. Therefore, for any $\boldsymbol{y}\in\mathbb{R}^{p}$ and $\theta>0,$ it holds that $\frac{\partial\|\text{Prox}_{\sigma\boldsymbol{\lambda}/\theta}(\boldsymbol{y})\|^{2}}{\partial\theta}=\frac{2}{\theta^{2}}\boldsymbol{1}^{\top}\text{Prox}_{\sigma\boldsymbol{\lambda}/\theta}(|\boldsymbol{y}|)$ and $\frac{\partial\boldsymbol{y}^{\top}\text{Prox}_{\sigma\boldsymbol{\lambda}/\theta}(\boldsymbol{y})}{\partial\theta}=\frac{1}{\theta^{2}}\sum_{i=1}^{p}|y_{i}|\mathbb{I}_{[\text{Prox}_{\sigma\boldsymbol{\lambda}/\theta}(\boldsymbol{y})]_{i}\neq 0}$ by chain rule. For $\theta=0$, we need to study the behavior of $\|\text{Prox}_{\sigma\boldsymbol{\lambda}/\theta}(\boldsymbol{y})\|$ when $\theta$ is closed to 0. It can be shown (for a proof, see (A.11) in [7])

[TABLE]

where $C_{\boldsymbol{\lambda}}=\{\boldsymbol{\nu}\in\mathbb{R}^{p}\mid\boldsymbol{\nu}\prec\boldsymbol{\lambda}\}$ (“$\prec$” denotes majorization, see Definition 2) is the unit ball of the dual norm of $J_{\boldsymbol{\lambda}}$ [9, Proposition 1.1] and $\text{Proj}_{C_{\boldsymbol{\lambda}}}$ is the orthogonal projection onto $C_{\boldsymbol{\lambda}}$. Take $\overline{\lambda}=\frac{1}{p}\sum_{i=1}^{p}\lambda_{i}$ and it is not hard to show $\overline{\boldsymbol{\lambda}}:=\overline{\lambda}\boldsymbol{1}_{p}$ satisfies $\overline{\boldsymbol{\lambda}}\prec\boldsymbol{\lambda}$. Clearly, $\frac{\sigma}{\theta}\overline{\boldsymbol{\lambda}}\prec\frac{\sigma}{\theta}\boldsymbol{\lambda}$ for $\sigma,\theta>0$ and $C_{\sigma\overline{\boldsymbol{\lambda}}/\theta}\subset C_{\sigma\boldsymbol{\lambda}/\theta}$, so from (187) we have

[TABLE]

Since $\overline{\lambda}>0$ (due to $\boldsymbol{\lambda}\neq\boldsymbol{0}$), (188) indicates that when $0<\theta\leq\frac{\sigma\overline{\lambda}}{\max(\{|y_{i}|\},1)}$, $\text{Prox}_{\sigma\boldsymbol{\lambda}/\theta}(\boldsymbol{y})=\boldsymbol{0}$. On the other hand, we let $\text{Prox}_{\sigma\boldsymbol{\lambda}/\theta}(\boldsymbol{y}):=\boldsymbol{0}$, when $\boldsymbol{\lambda}\neq\boldsymbol{0}$ and $\theta=0$. As a result, $\text{Prox}_{\sigma\boldsymbol{\lambda}/\theta}(\boldsymbol{y})=\boldsymbol{0}$ for \theta\in\big{[}0,\frac{\sigma\overline{\lambda}}{\max(\{|y_{i}|\},1)}\big{]} and combining the partial derivatives of $\|\text{Prox}_{\sigma\boldsymbol{\lambda}/\theta}(\boldsymbol{y})\|^{2}$ and $\boldsymbol{y}^{\top}\text{Prox}_{\sigma\boldsymbol{\lambda}/\theta}(\boldsymbol{y})$ on $\theta>0$ obtained above, we can get (183) and (184). ∎

-L Proof of Proposition 3

It directly follows from Proposition 1 that $\mathcal{M}_{\mu_{Y}}\subseteq\mathcal{I}$, so we just need to show $\mathcal{I}\subseteq\mathcal{M}_{\mu_{Y}}$.

For any $f\in\mathcal{I}$, consider the function $r(y)=y-f(y)$. It can be easily verified that $r(y)\in\mathcal{I}$. We claim that if we choose $\Lambda\sim r(|Y|)$ with $Y\sim\mu_{Y}$, then $f$ is the optimal solution of (6) (when $\tau=1$). Indeed, when $\tau=1$ and $\Lambda\sim r(|Y|)$, (6) can be equivalently written as

[TABLE]

where (a) follows from $g\in\mathcal{I}$ and in (b) we substitute $r(y)=y-f(y)$ and use the fact that $r\in\mathcal{I}$. From (189), we can see $f$ is the optimal solution of (6). On the other hand, since $f\in\mathcal{I}$ and $\mathbb{E}Y^{2}<\infty$, we can verify that $\mathbb{E}\Lambda^{2}<\infty$ and hence $\Lambda\in\mathcal{P}_{2}(\mathbb{R})$. In conclusion, for any $f\in\mathcal{I}$, we can always choose $\Lambda\sim|Y|-f(|Y|)$ satisfying $\Lambda\in\mathcal{P}_{2}(\mathbb{R})$, such that $f$ is the optimal solution of (6) (when $\tau=1$). By Proposition 1, this means $f(y)=\eta(y;\mu_{Y},\mu_{\Lambda})$ and hence $f\in\mathcal{M}_{\mu_{Y}}$. Therefore, $\mathcal{I}\subseteq\mathcal{M}_{\mu_{Y}}$.

-M Auxiliary Results for Proving Proposition 4

Lemma 18

For any $\sigma>0$, we have: (I) optimization problem (28) is convex and always has a unique optimal solution $f_{\sigma}\in\mathcal{I}$, (II) ${\cal L}(\sigma)$ defined in (28) is continuous at $\sigma$.

Proof:

(I) Optimization problem (28) can be equivalently written as:

[TABLE]

Then by the same arguments in the proof of Lemma 8, it is not hard to check for any $\sigma>0$, it is (strongly) convex and has a unique solution $f_{\sigma}\in\mathcal{I}$.

(II) Next, we prove the continuity of ${\cal L}(\sigma)$ at any $\sigma>0$. Define the following set

[TABLE]

where $H\sim\mathcal{N}(0,1)$ and $B\sim\mu_{B}$ are independent. Note that for any $\sigma>0$, we have $\mathcal{I}_{\sigma}\neq\emptyset$, since $\{f=0\}\in\mathcal{I}_{\sigma}$.

The first step is to show for any $\sigma,r>0$, there exists $\varepsilon\in(0,\sigma/2)$ such that whenever $\widehat{\sigma}\in\mathcal{B}_{\varepsilon}(\sigma)$ and $\widehat{f}\in\mathcal{I}_{\hat{\sigma}}$, we can always find a $f\in\mathcal{I}_{\sigma}$ satisfying $|f(x)-\widehat{f}(x)|<r$ almost everywhere on $\mathbb{R}$. This can be proved as follows. If $\widehat{f}\in\mathcal{I}_{\sigma}$, we can choose $f=\widehat{f}$, which trivially satisfies $|f(x)-\widehat{f}(x)|<r$; if $\widehat{f}\notin\mathcal{I}_{\sigma}$, then $\mathbb{E}\widehat{f}^{\prime}(B+\sigma H)>\delta$. Since $\hat{f}\in\mathcal{I}_{\hat{\sigma}}\subseteq\mathcal{I}$, it follows that $|\widehat{f}^{\prime}|\leq 1$ almost everywhere on $\mathbb{R}$. Meanwhile, since $\sigma,\widehat{\sigma}>0$, by the properties of Gaussian convolution we know both $B+\hat{\sigma}H$ and $B+\sigma H$ have smooth density functions supported on $\mathbb{R}$. Denote their density functions as $q_{1}$ and $q_{2}$. Then we have

[TABLE]

where $\text{TV}(\cdot,\cdot)$ and $\text{KL}(\cdot,\cdot)$ denote the total variation distance and KL-divergence between two probability measures and $C>0$ is a fixed constant only depending on $\sigma$. In reaching (191), (b) follows from Pinsker’s inequality and (c) follows from standard results of KL-divergence between Gaussian random variables and the fact that $\varepsilon\in(0,\sigma/2)$ and $\widehat{\sigma}\in\mathcal{B}_{\varepsilon}(\sigma)$. Step (a) in (191) can be obtained as follows. Recall that $\Pi(\mu_{1},\mu_{2})$ denotes the set of all couplings between measures $\mu_{1}$ and $\mu_{2}$. Then for any $(\widehat{\sigma}H_{1},\sigma H_{2})\in\Pi(\mu_{\widehat{\sigma}H},\mu_{\sigma H})$ and $B_{0}\sim\mu_{B}$ independent of $(\widehat{\sigma}H_{1},\sigma H_{2})$, we have $(B_{0}+\widehat{\sigma}H_{1},B_{0}+\sigma H_{2})\in\Pi(\mu_{B+\widehat{\sigma}H},\mu_{B+\sigma H})$. Therefore,

[TABLE]

where (a) and (b) follow from Strassen’s Theorem [56, p.7]. From (191), when $\varepsilon\in(0,\sigma/2)$ and $\widehat{\sigma}\in\mathcal{B}_{\varepsilon}(\sigma)$,

[TABLE]

Since $\widehat{f}\in\mathcal{I}_{\hat{\sigma}}$, from (190) and (193) we get

[TABLE]

We now slightly shrink $\widehat{f}$ to obtain the $f\in\mathcal{I}_{\sigma}$ satisfying $|f(x)-\widehat{f}(x)|<r$ almost everywhere. On one hand, we know $B+\sigma H$ has a density function supported on $\mathbb{R}$ for $\sigma>0$. On the other hand, since $\mathbb{E}\widehat{f}^{\prime}(B+\sigma H)>\delta$ (due to $\widehat{f}\notin\mathcal{I}_{\sigma}$) and $|\widehat{f}^{\prime}|\leq 1$ almost everywhere, it is not hard to show if $\varepsilon\leq\frac{\delta}{2C}$, there exists some $\mathcal{A}\subset\mathbb{R}_{>0}$ such that $\mathbb{E}[\widehat{f}^{\prime}(Y)\mathbb{I}_{Y\in\mathcal{A}}]=C\varepsilon$. Accordingly, we set

[TABLE]

and choose $f(y)$ to be:

[TABLE]

With this choice, from (194) we have $\mathbb{E}f^{\prime}(Y)\leq\delta-C\varepsilon$ and thus $f\in\mathcal{I}_{\sigma}$. In addition, $0\leq\widehat{f}(x)-f(x)\leq 2C\varepsilon$ almost everywhere. Hence we can choose $\varepsilon\leq\frac{r}{3C}$ so that $0\leq\widehat{f}(x)-f(x)<r$ almost everywhere. Summing up, for any $\sigma,r>0$, there exists $C$ depending only on $\sigma$ such that whenever $\varepsilon\leq\min\{\frac{r}{3C},\tfrac{\sigma}{2}\}$, $\widehat{\sigma}\in\mathcal{B}_{\varepsilon}(\sigma)$ and $\widehat{f}\in\mathcal{I}_{\hat{\sigma}}$, we can always find a $f\in\mathcal{I}_{\sigma}$ satisfying $0\leq\widehat{f}(x)-f(x)<r$ almost everywhere on $\mathbb{R}$.

Define ${\cal L}(f,\sigma)\overset{\text{def}}{=}\mathbb{E}[f(B+\varsigma H)-B]^{2}$, which is the objective function in (28). For any compact interval $I_{B}\subseteq\mathbb{R}_{>0}$ and any $\sigma_{1},\sigma_{2}\in I_{B}$, $f\in\mathcal{I}$, we have

[TABLE]

where $C_{1}$ is a constant that only depends on $I_{B}$. Therefore, for any $f\in\mathcal{I}$, $\mathcal{L}(f,\sigma)$ is uniformly Lipschitz continuous w.r.t. $\sigma$ on $I_{B}$. Consider $f_{\hat{\sigma}}$ which is the optimal solution of (28) under $\hat{\sigma}$. From the discussion in the last paragraph, for any $\sigma\in I_{B}$ and $r>0$, if $\widehat{\sigma}\in\mathcal{B}_{\varepsilon}(\sigma)\bigcap I_{B}$ under small enough $\varepsilon$, then there exists some $f\in\mathcal{I}_{\sigma}$ satisfying $|f_{\hat{\sigma}}(x)-f(x)|<r$ almost everywhere on $\mathbb{R}$. Therefore, for this $f$ we have

[TABLE]

where $C_{2}$ is some constant that does not depend on $\varepsilon,r$ and in the last step we use (196) and the fact that $|f_{\hat{\sigma}}(x)-f(x)|<r$ almost everywhere. Since $\mathcal{L}(\widehat{\sigma})=\mathcal{L}(f_{\hat{\sigma}},\widehat{\sigma})$, we have from (197)

[TABLE]

where (a) follows from (197) and (b) follows from definition of $\mathcal{L}(\sigma)$ in (28). By exchanging $\sigma$ and $\hat{\sigma}$, we can also get $\mathcal{L}(\sigma)\geq\mathcal{L}(\widehat{\sigma})-C_{2}r$. In conclusion, for any compact interval $I_{B}\subseteq\mathbb{R}_{>0}$ and $r>0$, there exists $\varepsilon>0$ such that for any $\sigma,\hat{\sigma}\in I_{B}$ satisfying $|\hat{\sigma}-\sigma|\leq\varepsilon$, we have $|\mathcal{L}(\widehat{\sigma})-\mathcal{L}(\sigma)|\leq r$. This proves the continuity of $\mathcal{L}(\sigma)$ over $\mathbb{R}_{>0}$. ∎

Lemma 19

Equation (29) satisfies the following: (I) it always has a solution and the minimum solution $\sigma_{0}\in[\sigma_{w},(\sigma_{w}^{2}+\delta^{-1}\mathbb{E}B^{2})^{1/2}]$, (II) $\sigma_{0}=\inf\mathcal{A}$, where set $\mathcal{A}$ is defined in (70).

Proof:

(I) We first prove $\sigma_{0}$ always exists and $\sigma_{0}\in I_{B}$, where $I_{B}:=[\sigma_{w},(\sigma_{w}^{2}+\delta^{-1}\mathbb{E}B^{2})^{1/2}]$. It is not hard to verify that at the boundary of $I_{B}$, ${\cal L}(\sigma)$ satisfies

[TABLE]

Indeed, when \sigma=\big{(}\sigma_{w}^{2}+\tfrac{\mathbb{E}B^{2}}{\delta}\big{)}^{1/2}, set $f=0$ in (28) and $\mathbb{E}[f(B+\sigma H)-B]^{2}=\mathbb{E}B^{2}=\delta(\sigma^{2}-\sigma_{w}^{2})$, so ${\cal L}(\sigma)\leq\delta(\sigma^{2}-\sigma_{w}^{2})$. On the other hand, since ${\cal L}(\sigma)\geq 0$, the second inequality of (198) immediately follows. Then by (198), the continuity of ${\cal L}(\sigma)$ shown in Lemma 18 and the fact that $\sigma_{0}\geq\sigma_{w}$, we know $\sigma_{0}$ always exists and $\sigma_{0}\in I_{B}$.

(II) To prove $\inf\mathcal{A}=\sigma_{0}$, we proceed as follows:

1. (i)

Show

[TABLE] 2. (ii)

Prove the following membership certificate of set $\mathcal{A}$ for those $\sigma\in I_{B}$:

[TABLE]

It is not hard to see the above steps will imply $\inf\mathcal{A}=\sigma_{0}$. Indeed, combining (200) with (199) yields the following characterization of $\inf\mathcal{A}$:

[TABLE]

By (198) and the continuity of ${\cal L}(\sigma)$, the infimum on the RHS of (201) is reached by $\sigma_{0}$, which always exists. Therefore, $\inf\mathcal{A}=\sigma_{0}$. Step (i)-(ii) can be proved as follows:

[Proof of (i)] From the first equation of (69), we have $\inf\mathcal{A}\geq\sigma_{w}$. On the other hand, since $f=0$, \sigma=\big{(}\sigma_{w}^{2}+\tfrac{\mathbb{E}B^{2}}{\delta}\big{)}^{1/2}, $\tau=1$ is a solution of (66)-(67), from (68) we have \sigma_{\text{opt}}\leq\big{(}\sigma_{w}^{2}+\tfrac{\mathbb{E}B^{2}}{\delta}\big{)}^{1/2}. This together with the lower bound of $\sigma_{\text{opt}}$ in (71) indicates: \inf\mathcal{A}\leq\sigma_{\text{opt}}\leq\big{(}\sigma_{w}^{2}+\tfrac{\mathbb{E}B^{2}}{\delta}\big{)}^{1/2}. Therefore, $\inf\mathcal{A}\in I_{B}$.

[Proof of (ii)] The “$\Rightarrow$” direction of (200) immediately follows from the definition of $\mathcal{A}$ in (70). For the other direction, suppose we have a $\sigma\in I_{B}$ satisfying ${\cal L}(\sigma)\leq\delta(\sigma^{2}-\sigma_{w}^{2})$. Then

[TABLE]

where the first equality is due to that ${\cal L}(\sigma)$ can be achieved by $f_{\sigma}$. Now consider the shrinkage of $f_{\sigma}$ as: $\alpha f_{\sigma}$, where $\alpha\in[0,1]$. Clearly, for any $\alpha\in[0,1]$, $\alpha f_{\sigma}$ still satisfies $\mathbb{E}[\alpha f_{\sigma}^{\prime}(B+\sigma H)]\leq\delta$. Also we have:

[TABLE]

where the last inequality follows from the condition that $\sigma\in I_{B}$. On the other hand, it can be easily checked that $\alpha\mapsto\mathbb{E}[\alpha f_{\sigma}(B+\sigma H)-B]^{2}$ is continuous, so from (203), (202), there exists $\alpha_{0}\in[0,1]$ such that $(\alpha_{0}f_{\sigma},\sigma)$ is a solution of (69), indicating $\sigma\in\mathcal{A}$. ∎

-N Auxiliary Results for Proving Proposition 5

Lemma 20

For a probability measure $\mu_{Y}\in\mathcal{P}_{2}(\mathbb{R})$, define

[TABLE]

where q_{0}\overset{\text{def}}{=}\mathbb{P}\big{(}\eta(Y;\mu_{Y},\mu_{\Lambda})=0\big{)}. Then for any $\mu_{Y}\in\mathcal{P}_{2}(\mathbb{R})$, we have $\widetilde{\mathcal{M}}_{\mu_{Y}}=\mathcal{I}$. Correspondingly, for any $f(y)\in\mathcal{I}$, we can take $\mu_{\Lambda}$ as the law of $\max\{y_{0},|Y|-f(|Y|)\}$ ($Y\sim\mu_{Y}$), so that $\eta(y\,;\mu_{Y},\mu_{\Lambda})=f(y)$. Here $y_{0}\overset{\text{def}}{=}\sup_{y\geq 0}\left\{y\mid f(y)=0\right\}$,

Proof:

The proof is similar as Proposition 3. When $\mu_{\Lambda}$ is the law of $\max\{y_{0},|Y|-f(|Y|)\}$, optimization (6) ($\tau=1$) becomes:

[TABLE]

Since the feasible set in (205) is $\mathcal{I}$, we know the optimal solution of (205) is exactly $f$. Recall that $\eta(y;\mu_{Y},\mu_{\Lambda})$ is the optimal solution of (6) ($\tau=1$), so we have $\eta(y;\mu_{Y},\mu_{\Lambda})=f(y)$.

Finally, we show the law of $\max\{y_{0},|Y|-f(|Y|)\}$ satisfies the constraint in (204). Since $\mu_{Y}\in\mathcal{P}_{2}(\mathbb{R})$ and $f\in\mathcal{I}$, we can easily get $\mu_{\Lambda}\in\mathcal{P}_{2}(\mathbb{R})$. On the other hand, suppose $q_{0}>0$. For any $t\in[0,q_{0})$, we have

[TABLE]

where in (a) we use the fact that $F_{|Y|}^{-1}(q_{0})\leq y_{0}$, since $\eta(y;\mu_{Y},\mu_{\Lambda})=f(y)$ and q_{0}=\mathbb{P}\big{(}\eta(Y;\mu_{Y},\mu_{\Lambda})=0\big{)}, (b) follows from $F_{|Y|}^{-1}(t)<F_{|Y|}^{-1}(q_{0})=y_{0}$, since $t<q_{0}$ and (c) is due to our choice of $\Lambda$, which yields $F_{\Lambda}^{-1}(u)=y_{0}$ for any $0\leq u\leq q_{0}$. ∎

Lemma 21

For $\alpha\in[0,1]$ and $\sigma>0$, we have: (I) optimization problem (41) is convex and always has a unique optimal solution $f_{\alpha,\sigma}\in\mathcal{I}$, (II) $\mathcal{L}_{\alpha}(\sigma)$ defined in (41) is a continuous function over $\mathbb{R}_{>0}$, (III) equation $\mathcal{L}_{\alpha}(\sigma)=\delta(\sigma^{2}-\sigma_{w}^{2})$ always has a solution and the minimum solution \sigma_{0,\alpha}\in\big{[}\sigma_{w},\,\sqrt{\sigma_{w}^{2}+\delta^{-1}\mathbb{E}B^{2}}\big{]}.

Proof:

(I) Comparing with (28) and (41), we can find the only difference is that in (41), we add a constraint $f\in\mathcal{F}_{\alpha,\sigma}$. It is not hard to check for any $\alpha\in[0,1]$ and $\sigma>0$, the set $\mathcal{F}_{\alpha,\sigma}$ is convex and closed in the $L^{2}$ space $\mathcal{H}_{\mu_{Y}}$ [definition can be found in (84)], so the uniqueness of optimal solution of (41) still holds using the same arguments.

(II) The case of $\alpha=0\text{ or }1$ is easy. When $\alpha=1$, we have $\mathcal{I}\subseteq\mathcal{F}_{\alpha,\sigma}$, so $\mathcal{L}_{\alpha}(\sigma)=\mathcal{L}(\sigma)$ and its continuity is proved in the last part of Lemma 18; when $\alpha=0$, $\mathcal{F}_{\alpha,\sigma}$ contains only one element: $f(x)=0$ and $\mathcal{L}_{\alpha}(\sigma)=\mathbb{E}B^{2}$, which is trivially continuous. Therefore, it only remains to verify for the case when $\alpha\in(0,1)$.

The proof for case $\alpha\in(0,1)$ is similar to the proof of continuity of $\mathcal{L}(\sigma)$ in Lemma 18. For any $\alpha\in(0,1)$ and $\sigma>0$, define the following set

[TABLE]

where $H\sim\mathcal{N}(0,1)$ and $B\sim\mu_{B}$ are independent. We always have $\mathcal{I}_{\alpha,\sigma}\neq\emptyset$, since $\{f=0\}\in\mathcal{I}_{\alpha,\sigma}$. Next we show that for any $\alpha\in(0,1)$ and $\sigma,r>0$, there exists $\varepsilon\in(0,\sigma/2)$ such that whenever $\widehat{\sigma}\in\mathcal{B}_{\varepsilon}(\sigma)$ and $\widehat{f}\in\mathcal{I}_{\alpha,\hat{\sigma}}$, we can always find a $f\in\mathcal{I}_{\alpha,\sigma}$ satisfying $|f(x)-\widehat{f}(x)|<r$ almost everywhere on $\mathbb{R}$. First, for any $\hat{\sigma}>0$ and $\widehat{f}\in\mathcal{I}_{\alpha,\hat{\sigma}}$, we have $|\widehat{f}(y)|=0$, when $|y|\leq\Phi^{-1}(1-\tfrac{\alpha}{2})\hat{\sigma}$. We can then apply the following shrinkage to $\widehat{f}$:

[TABLE]

One can check $\widehat{f}_{T}$ in (207) satisfies: $\widehat{f}_{T}\in\mathcal{I}_{\alpha,\hat{\sigma}}\cap\mathcal{F}_{\alpha,\sigma}$ and

[TABLE]

almost everywhere on $\mathbb{R}$. For small enough $\varepsilon>0$ and $\widehat{\sigma}\in\mathcal{B}_{\varepsilon}(\sigma)$, we can then follow the same steps leading to (195) in the proof of continuity of $\mathcal{L}(\sigma)$ in Lemma 18 to obtain a $f\in\mathcal{I}$ satisfying

[TABLE]

almost everywhere on $\mathbb{R}$ and $\mathbb{E}f^{\prime}(B+\sigma H)\leq\delta$. Meanwhile, since $\widehat{f}_{T}\in\mathcal{F}_{\alpha,\sigma}$, we also have $f\in\mathcal{F}_{\alpha,\sigma}$. As a result, $f\in\mathcal{I}_{\alpha,\sigma}$. Besides, combining (208) and (209) we have for small enough $\varepsilon>0$ and $\widehat{\sigma}\in\mathcal{B}_{\varepsilon}(\sigma)$, $0\leq\hat{f}(y)-f(y)\leq r$ almost everywhere on $\mathbb{R}$. The remaining proof is completely same as the last part of the proof of $\mathcal{L}(\sigma)$’s continuity and we omit the details here.

(III) The proof is the same as Lemma 19. Using the same argument, it can be verified that $\mathcal{L}_{\alpha}(\sigma)$ also satisfies

[TABLE]

Then by the continuity of $\mathcal{L}_{\alpha}(\sigma)$ and the fact that $\sigma_{0,\alpha}\geq\sigma_{w}$, we get the desired result. ∎

Lemma 22

For any given $\alpha\in[0,1]$, we have the following.

1. (a)

It is always true that $\sigma_{\text{opt},\alpha}\geq\sigma_{0,\alpha}$. 2. (b)

If \delta^{-1}\mathbb{E}\big{[}f_{\alpha}^{\prime}(Y_{0,\alpha})\big{]}<1, then $\sigma_{\text{opt},\alpha}=\sigma_{0,\alpha}$ and the infimum of (73) can be achieved by choosing $\mu_{\Lambda}=\mu_{\text{opt},\alpha}$.

Proof:

The proof is similar to that of Proposition 4 (c). Recall that a key step in the proof is the conversion of the optimization over $\mu_{\Lambda}$ into an equivalent optimization over realizable limiting scalar functions $f$. We want to adopt the same strategy here, but since an additional constraint on $\mu_{\Lambda}$ is added, we need to determine the new realizable set of $f$, as we did in Proposition 3. It turns out that the new realizable set is still $\mathcal{I}$. This result is proved in In Lemma 20 and it enables us to follow the same steps leading to (68) to show that (73) is equivalent to

[TABLE]

where $\mathcal{D}_{F}(\alpha)$ is defined as:

[TABLE]

Comparing (68) and (210), it can be seen that the only difference is that in (210) we require $f\in\mathcal{F}_{\alpha,\sigma}$, which is needed to control the type-I error level. Then similar as (70) we define

[TABLE]

and it holds that $\sigma_{\text{opt},\alpha}\geq\inf\mathcal{A}(\alpha)$. Meanwhile, by the same reasoning in Lemma 19, we can also show $\inf\mathcal{A}(\alpha)=\sigma_{0,\alpha}$. This gives us $\sigma_{\text{opt},\alpha}\geq\sigma_{0,\alpha}$.

Now consider the scenario where \delta^{-1}\mathbb{E}\big{[}f_{\alpha}^{\prime}(Y_{0,\alpha})\big{]}<1. In this case, we have $\tau_{0,\alpha}\in(0,\infty)$. Then it is not hard to check $(f_{\alpha},\sigma_{0,\alpha},\tau_{0,\alpha})$ satisfies equation (66)-(67). Therefore, $(\sigma_{0,\alpha},\tau_{0,\alpha})\in\mathcal{D}_{F}(\alpha)$. By (210), we have $\sigma_{0,\alpha}\geq\sigma_{\text{opt},\alpha}$. Since we already get $\sigma_{\text{opt},\alpha}\geq\sigma_{0,\alpha}$, we can conclude that $\sigma_{\text{opt},\alpha}=\sigma_{0,\alpha}$. Let us denote $(\sigma_{*},\tau_{*})$ as the solution of fixed-point equation (13)-(14), when $\mu_{\Lambda}=\mu_{\text{opt},\alpha}$. Using Lemma 20, it is not hard to check

[TABLE]

and

[TABLE]

Meanwhile, we have $\mu_{\text{opt},\alpha}\in\widetilde{\mathcal{P}}_{\Lambda}$. Recall that the infimum of $\sigma_{*}$ in (73) is $\sigma_{0,\alpha}$ [c.f. (210)]. As a result, the infimum of $\sigma_{*}$ in (73) is reached when $\mu_{\Lambda}=\mu_{\text{opt},\alpha}$. ∎

Lemma 23

For $\alpha\in(0,1]$, if \delta^{-1}\mathbb{E}\big{[}f_{\alpha}^{\prime}(Y_{0,\alpha})\big{]}<1 and $y_{0,\alpha}=\Phi^{-1}(1-\frac{\alpha}{2})\sigma_{0,\alpha}$, then $\overline{\mathcal{\mathcal{P}}}(\alpha)=\mathcal{P}(\alpha)$ and $\lim_{p\to\infty}\text{Power}=\mathcal{P}(\alpha)$, when $\mu_{\Lambda}=\mu_{\text{opt},\alpha}$.

Proof:

In the proof, let $(\sigma_{*},\tau_{*})$ be the solution of fixed-point equation (13)-(14), when $\mu_{\Lambda}=\mu_{\text{opt},\alpha}$. Also denote $y_{\text{th}}^{*}:=\sup_{y\geq 0}\{y\mid\eta(y;\mu_{Y_{*}},\mu_{\tau_{*}\Lambda})=0\}$.

Assume \delta^{-1}\mathbb{E}\big{[}f_{\alpha}^{\prime}(Y_{0,\alpha})\big{]}<1 and $y_{0,\alpha}=\Phi^{-1}(1-\frac{\alpha}{2})\sigma_{0,\alpha}$. Recall from the proof of Lemma 22, when $\mu_{\Lambda}=\mu_{\text{opt},\alpha}$, we have

[TABLE]

where (a) follows from (212), (b) follows from assumption $y_{0,\alpha}=\Phi^{-1}(1-\frac{\alpha}{2})\sigma_{0,\alpha}$ and (c) follows from (211). Therefore,

[TABLE]

where the last equality is due to that $\mu_{\text{opt},\alpha}$ is the optimal solution of (73), as is proved by Lemma 22. Therefore, from (213) we know when $\mu_{\Lambda}=\mu_{\text{opt},\alpha}$, the objective value of (40) equals to $\overline{\mathcal{\mathcal{P}}}(\alpha)$. This implies $\mathcal{P}(\alpha)\geq\overline{\mathcal{\mathcal{P}}}(\alpha)$, since $\mathcal{P}(\alpha)$ is the optimal value of (40). Combined with the fact that $\overline{\mathcal{\mathcal{P}}}(\alpha)$ is the upper bound of $\mathcal{P}(\alpha)$ [c.f. (76)], it then follows that $\overline{\mathcal{\mathcal{P}}}(\alpha)=\mathcal{P}(\alpha)$. Also when $\mu_{\Lambda}=\mu_{\text{opt},\alpha}$, $\lim_{p\to\infty}\text{Power}=\overline{\mathcal{\mathcal{P}}}(\alpha)=\mathcal{P}(\alpha)$.

∎