Approximate separability of symmetrically penalized least squares in high dimensions: characterization and consequences

TL;DR

This paper demonstrates that symmetrically penalized least squares in high dimensions behave similarly to separable penalties, with implications for adaptive procedures and the role of non-separability in high-dimensional estimation.

Contribution

It provides a precise quantification of the near-equivalence between symmetric and separable penalties in high-dimensional models, clarifying the impact of non-separability.

Findings

High-dimensional behavior of symmetric penalties closely matches separable penalties.

Limited advantages of non-separable penalties when the empirical distribution is known.

Non-separable penalties automatically implement adaptive procedures when the distribution is unknown.

Abstract

We show that the high-dimensional behavior of symmetrically penalized least squares with a possibly non-separable, symmetric, convex penalty in both (i) the Gaussian sequence model and (ii) the linear model with uncorrelated Gaussian designs nearly matches the behavior of least squares with an appropriately chosen separable penalty in these same models. The similarity in behavior is precisely quantified by a finite-sample concentration inequality in both cases. Our results help clarify the role non-separability can play in high-dimensional M-estimation. In particular, if the empirical distribution of the coordinates of the parameter is known --exactly or approximately-- there are at most limited advantages to using non-separable, symmetric penalties over separable ones. In contrast, if the empirical distribution of the coordinates of the parameter is unknown, we argue that non-separable, symmetric penalties automatically implement an adaptive procedure which we characterize. We also provide a partial converse which characterizes adaptive procedures which can be implemented in this way.

Equations

$$\mathbf{y}=\mathbf{\theta }+\tau \mathbf{z},$$

$$\mathbf{y}=\mathbf{X}\mathbf{\theta }+\mathbf{w},$$

$$\mathsf{prox}[f_p](\mathbf{y})=\arg \underset{\mathbf{x}\in {\mathbb{R}}^p}{\min }\left\{\frac{1}{2}\Vert \mathbf{y}-\mathbf{x}{\Vert }^2+f_p(\mathbf{x})\right\}.$$

$$\hat{\mathbf{\theta }}\in \arg \underset{\mathbf{b}\in {\mathbb{R}}^p}{\min }\left\{\frac{1}{2n}\Vert \mathbf{y}-\mathbf{X}\mathbf{b}{\Vert }^2+f_p(\mathbf{b})\right\},$$

$$f_p(\mathbf{x})=\sum _{j=1}^p\rho (x_j),$$

$$\mathsf{prox}[f_p](\mathbf{y}{)}_j=\mathsf{prox}[\rho ](y_j),$$

$$f_p(\mathbf{x})$$

$$\mathbf{x}\mapsto \frac{1}{2}\left(F(|\mathsf{supp}(\mathbf{x})|)+\Vert \mathbf{x}{\Vert }_2^2\right),$$

$$f_p(\mathbf{x})=\sum _{j=1}^p{\lambda }_j|x{|}_{(j)}$$

$$\mathbb{P}\left(W_2({\mu }_{(\hat{\mathbf{\theta }},\mathbf{\theta })},{\mu }_{\ast }{)}^2\ge ϵ\right)\le C{ϵ}^{-3}\exp \left(-cp{ϵ}^3\log (1/ϵ{)}^{-2}\right),$$

$$W_2(\mu ,{\mu }^{\prime }{)}^2:=\underset{\pi \in \Pi (\mu ,{\mu }^{\prime })}{\inf }{\mathbb{E}}_{(X,X^{\prime })\sim \pi }\mathbb{E}[(X-X^{\prime }{)}^2]$$

$${\mathcal{PR}}_1:=\{\mathsf{prox}[\rho ]\mid \rho :\mathbb{R}\to \bar{\mathbb{R}}\text{ lsc, proper, convex}\}.$$

$$\mathbb{P}\left(\frac{1}{p}\Vert \mathsf{prox}[f_p](\mathbf{\theta }+\tau \mathbf{z})-{\mathfrak{A}}_{f_p}({\mu }^{\ast \tau })(\mathbf{\theta }+\tau \mathbf{z}){\Vert }^2>4(W_2(\mu ,{\mu }_{\mathbf{\theta }})+\tau \sqrt{ϵ}{)}^2\right)$$

$$\le C{ϵ}^{-1}\exp (-cp{ϵ}^3\log (ϵ{)}^{-2}),$$

$$\mathbb{P}\left(\frac{1}{p}\Vert \mathsf{prox}[f](\mathbf{\theta }+\tau \mathbf{z})-{\mathfrak{A}}_{f_p}({\mu }_{\mathbf{\theta }}^{\ast \tau })(\mathbf{\theta }+\tau \mathbf{z}){\Vert }^2>ϵ\right)\le C{ϵ}^{-1}\exp (-cp{ϵ}^3\log (ϵ{)}^{-2}).$$

$$\mathcal{C}=\left\{\{f_p:{\mathbb{R}}^p\to \bar{\mathbb{R}}\}:f_p\text{ is lsc, proper, symmetric, and convex ∀p}\right\}$$

$${\mathsf{R}}_{\mathsf{symm}}^{\mathsf{opt}}(\tau ;\mu )=\underset{\{f_p\}\in \mathcal{C}}{\inf }\underset{p\to \infty }{\mathrm{liminf}}\frac{1}{p}\mathbb{E}\left[\Vert \mathsf{prox}[f_p](\mathbf{\theta }+\tau \mathbf{z})-\mathbf{\theta }{\Vert }^2\right],$$

$$\tau_{\mathsf{symm}}^{2}=\sup\left\{\tau^{2}\Biggm{|}\delta(\tau^{2}-\sigma^{2})<\mathsf{R_{symm}^{opt}}(\tau;\mu)\right\}.$$

$${\mu }_{\mathbf{\theta }}\stackrel{\mathrm{W}}{\to }\mu ,\frac{1}{n}\Vert \mathbf{w}{\Vert }^2\to {\sigma }^2,$$

$$\underset{\{f_p\}\in {\mathcal{C}}_{\delta ,\mu }}{\inf }\underset{p\to \infty }{\overset{\mathrm{p}}{\mathrm{liminf}}}\frac{1}{p}\Vert \hat{\mathbf{\theta }}-\mathbf{\theta }{\Vert }^2\ge \delta ({\tau }_{\mathsf{symm}}^2-{\sigma }^2),$$

$$\begin{array}{c}\text{for all compact T⊂(0,∞), there exists λˉ=λˉ(T)<∞ such that}\\ \underset{p\to \infty }{\mathrm{limsup}}\underset{\lambda >\bar{\lambda },\tau \in T}{\sup }\frac{1}{\tau p}\mathbb{E}\left[⟨\mathbf{z},\mathsf{prox}[\lambda f_p](\mathbf{\theta }+\tau \mathbf{z})\right],\end{array}$$

$${\mathsf{R}}_{\mathsf{sep}}^{\mathsf{opt}}(\tau ;\mu )=\underset{f_1\in {\mathcal{F}}_1}{\inf }\mathbb{E}\left[(\mathsf{prox}[f_1](\theta +\tau z)-\theta {)}^2\right],$$

$$\tau_{\mathsf{sep}}^{2}=\sup\left\{\tau^{2}\Biggm{|}\delta(\tau^{2}-\sigma^{2})<\mathsf{R_{sep}^{opt}}(\tau;\mu)\right\}.$$

$$\underset{\begin{array}{c}{f}_1\in {\mathcal{F}}_1\\ \left\{f_p(\mathbf{x})={\sum }_{j=1}^p{f}_1(x_j)\right\}\in {\mathcal{C}}_{\delta ,\mu }\end{array}}{\inf }\underset{p\to \infty }{\overset{\mathrm{p}}{\mathrm{liminf}}}\frac{1}{p}\Vert \hat{\mathbf{\theta }}-\mathbf{\theta }{\Vert }^2\ge \delta ({\tau }_{\mathsf{sep}}^2-{\sigma }^2),$$

$${\mathsf{R}}_{\mathsf{symm}}^{\mathsf{opt}}(\tau ;\mu )={\mathsf{R}}_{\mathsf{sep}}^{\mathsf{opt}}(\tau ;\mu ).$$

$${{\tau }^{\ast }}^2$$

$$\delta$$

$$\mathbb{P}\left(W_2({\hat{\mu }}_{(\hat{\mathbf{\theta }},\mathbf{\theta })},{\mu }_{({\mathfrak{A}}_{{\lambda }^{\ast }{f}_p}({\mu }^{\ast {\tau }^{\ast }})\circ Y^{\ast },{\Theta }^{\ast })}{)}^2>{{\tau }^{\ast }}^2ϵ\right)$$

$$\le 2C{ϵ}^{-1}\exp \left(-c_{2}p{ϵ}^3\log (1/ϵ{)}^{-2}\right)+4\mathbb{P}\left(\left|\frac{\Vert \mathbf{w}\Vert /\sqrt{n}}{{\sigma }^{\ast }}-1\right|>c_3ϵ\right).$$

$$\sum _{j=1}^n\mathbb{E}[X_j{G}_j]\ge \sum _{j=1}^n\mathbb{E}[X_j{G}_{\sigma (j)}].$$

$$\mathbb{P}\left(\frac{1}{p}\Vert \mathsf{prox}[f_p](\mathbf{\theta }+\tau \mathbf{z})-\mathfrak{A}({\mu }^{\ast \tau })(\mathbf{\theta }+\tau \mathbf{z}){\Vert }^2>(W_2(\mu ,{\mu }_{\mathbf{\theta }})+\tau \sqrt{ϵ}{)}^2\right)$$

$$\le C{ϵ}^{-1}\exp (-cp{ϵ}^3\log (ϵ{)}^{-2}),$$

Full text

1 Introduction

In this paper, we consider estimation in two closely related statistical models. First, we consider the Gaussian sequence model

[TABLE]

where ${\boldsymbol{\theta}}\in\mathbb{R}^{p}$ is a parameter vector, possibly fixed or random, $\tau\geq 0$, and ${\boldsymbol{z}}\sim{\sf N}(0,I_{p})$. In the sequence model, the statistician observes ${\boldsymbol{y}}$ and estimates ${\boldsymbol{\theta}}$. Second, we consider the linear model

[TABLE]

where ${\boldsymbol{\theta}}\in\mathbb{R}^{p}$, ${\boldsymbol{w}}\in\mathbb{R}^{n}$, and $X_{ij}\stackrel{{\scriptstyle\mathrm{iid}}}{{\sim}}{\sf N}(0,1/n)$. In the linear model, the statistician observes ${\boldsymbol{y}}$ and ${\boldsymbol{X}}$ and wishes to estimate ${\boldsymbol{\theta}}$. The noise ${\boldsymbol{w}}$ may or may not be random, but we require that it be independent of the design ${\boldsymbol{X}}$.

M-estimation is a popular approach to estimation which involves solving a data-depending optimization problem. The M-estimators we consider minimize the sum of a least squares loss and a convex regularization penalty. In the sequence model (1.1), such M-estimators are commonly known as proximal operators. In particular, if $f_{p}:\mathbb{R}^{p}\rightarrow\bar{\mathbb{R}}$ is a lower semi-continuous (lsc), proper, convex function, the proximal operator $\mathsf{prox}[f_{p}]:\mathbb{R}^{p}\rightarrow\mathbb{R}^{p}$ is defined via

[TABLE]

In the linear model (1.2), such M-estimators are defined to satisfy

[TABLE]

where we specify set membership rather than equality because the minimizing set in (1.4), unlike that in (1.3), is not necessarily a singleton. When the statistician only has access to (or chooses only to exploit) structural information on the empirical distribution of the coordinates of ${\boldsymbol{\theta}}$ (and not the order in which these coordinates appear), it is natural to restrict attention to $f_{p}$ which are invariant to permuting the coordinates of its argument. Such $f_{p}$ we will call symmetric.

In this paper, we provide results which precisely characterize for any symmetric, convex $f_{p}$ the behavior of the estimators (1.3) and (1.4) in the models (1.1) and (1.2), respectively. Our results have several consequences on the design and potential use of such estimators, which we also describe. We begin by summarizing some of these results.

1.1 Penalized least squares in the sequence model

Consider that $f_{p}$ is both symmetric and separable; that is,

[TABLE]

where $\rho:\mathbb{R}\rightarrow\bar{\mathbb{R}}$ is convex. Then the proximal operator (1.3) satisfies

[TABLE]

for all $j$. Thus, observation $i$ does not affect estimate $j$ for $i\neq j$. This separability allows us to study statistical properties of the proximal operator in the sequence model by studying statistical properties of the scalar proximal operator $\mathsf{prox}[\rho]$ in the model $y=\theta+\tau z$ where $\theta\in\mathbb{R}$ and $z\sim\mathsf{N}(0,1)$.

A main insight of this paper is that in high-dimensions, all symmetric proximal operators are “approximately separable” in a sense which we make precise in Section 2. As a demonstration, consider the penalty

[TABLE]

where $\lambda_{1}\geq\lambda_{2}\geq\cdots\geq\lambda_{p}$ and $\eta_{(j)}$ denotes the $j^{\text{th}}$ decreasing order statistic of $\eta$. This penalty is convex, symmetric, and non-separable. It is a member of a large class of convex relaxations to submodular combinatorial penalties that are studied in detail by [OB12]. It has been referred to as a smoothed ordered weighed $\ell_{1}$ (OWL) norm by [SBB17]. One derivation of (1.7) is as the tightest positively homogeneous convex lower bound of the combinatorial penalty

[TABLE]

where $F(j)=\sum_{i=1}^{j}\lambda_{j}$ is a sub-modular function of the number of non-zero coordinates of ${\boldsymbol{x}}$ [OB12, Corollary1, Lemma 8]. The penalty (1.7) was further studied by [SBB17] in the context of sparse linear regression with correlated designs. We refer the reader to [OB12, SBB17] and references therein for a discussion of this penalty and its uses.

In Figure 1, we report the results of a simulation study in which we applied the proximal operator for penalty (1.7) in the sequence model (1.1) when the dimension $p=1000$; the parameter ${\boldsymbol{\theta}}$ has 50 coordinates equal to $-1$, 50 coordinates equal to 1, and 900 coordinates equal to 0; $\lambda_{j}=2$ for $j\leq 333$, $\lambda_{j}=1$ for $334\leq j\leq 667$, and $\lambda_{j}=.5$ for $668\leq j\leq 1000$; and $\tau$ is either $.5,1$, or $2.5$. For each $\tau=.5,1,2.5$, we randomly select some of the indices $j$ and display $(y_{j},\widehat{\theta}_{j})$ using red open triangles, green circles, and blue filled triangles, respectively. For each $\tau$, we also plot a curve which we have computed before the realization of the noise and on which our theory predicts all such pairs $(y_{j},\widehat{\theta}_{j})$ should approximately lie. We see that for a fixed value of $\tau$, each pair $(y_{j},\widehat{\theta}_{j})$ does indeed approximately lie on the corresponding pre-computed curve.

These observations suggest –and we will show– that conditional on (i) the empirical parameter distribution and (ii) the noise level, the estimate $\widehat{\theta}_{j}$ is approximately only a function of $y_{j}$. This function does not depend upon $j$ and can be determined prior to the realization of the noise. Moreover, we will show that this function can be written as $\mathsf{prox}[\rho]$ for some scalar convex function $\rho$. In this sense, for the purposes of estimation in the Gaussian sequence model with symmetrically penalized least squares, all penalties behave like separable penalties in high-dimensions. For statistical purposes, they are “approximately separable.” We make these results precise with a finite sample concentration inequality which holds uniformly over choices of symmetric penalty $f_{p}$. These results suggest that there are limited advantages to non-separability in the sequence model when the empirical parameter distribution is approximately known.

A recent paper established this phenomenon for a particular class of symmetric penalties: the ordered weighted $\ell_{1}$ (OWL) norms, which induce an estimation scheme also refered to as sorted $\ell_{1}$ penalized estimation (SLOPE) [HL19]. These penalties have been proposed for the purposes of adaptation to sparsity and greater stability in sparse estimation with highly correlated designs [BvdBS*+*15, SC16, SBB17]. They take the form

[TABLE]

where $\lambda_{1}\geq\cdots\geq\lambda_{p}\geq 0$ are appropriately chosen regularization parameters and $|x|_{(j)}$ are the decreasing absolute order statistics of ${\boldsymbol{x}}$. We establish approximate separability of symmetric proximal operators much more generally because our results hold for any symmetric penalty. For example, the theorems in [HL19] do not address the approximate separability of the penalty (1.7).

1.2 Penalized least squares in the linear model with Gaussian designs

In the past several years, there have been many works characterizing the distribution of M-estimators in high-dimensional linear models with Gaussian design matrices (see e.g. [BM12, EKBB*+*13, TOH15, TAH18, DM16, MM18]). To facilitate the discussion to come, we present a recent result of this type for the LASSO.

Theorem 1** (Adapted from Theorem 3.1 of [MM18]).**

Consider model (1.2) with ${\boldsymbol{w}}\sim\mathsf{N}({\boldsymbol{0}},\sigma^{2}{\boldsymbol{I}}_{n})$ for $\sigma>0$. Let $f_{p}({\boldsymbol{x}})=\xi\|{\boldsymbol{x}}\|_{1}$. Then there exist $\tau,\lambda,C,c$ depending only on the empirical distribution of the parameters $\mu_{{\boldsymbol{\theta}}}:=\frac{1}{p}\sum_{j=1}^{p}\delta_{\theta_{j}}$, the measurement rate $n/p$, the noise variance $\sigma^{2}$, and the regularization parameter $\xi$ such that for all $\epsilon\in(0,1/2)$

[TABLE]

where $\mu_{(\widehat{\boldsymbol{\theta}},{\boldsymbol{\theta}})}=\frac{1}{p}\sum_{j=1}^{p}\delta_{(\widehat{\theta}_{j},\theta_{j})}$ is the empirical joint distribution of the true parameters and their corresponding estimates; $\mu^{*}$ is the joint distribution of $(\eta_{\mathrm{soft}}(\theta+\tau z;\lambda\xi),\theta)$ when $\theta\sim\mu_{{\boldsymbol{\theta}}}$, $z\sim\mathsf{N}(0,1)$ independent of $\theta$, and $\eta_{\mathrm{soft}}$ is soft-thresholding with threshold $\lambda\xi$;111That is, $\eta_{\mathrm{soft}}(y;\lambda\xi)=\max(\min(y+\lambda\xi,0),y-\lambda\xi)$. and $W_{2}$ is the Wasserstein distance of order 2 on the space of probability distributions with finite second moment (which we define in Section 1.6).

Coordinate-wise soft-thresholding with threshold $\lambda\xi$ is the proximal operator of the separable penalty $f_{p}({\boldsymbol{x}})=\lambda\xi\|{\boldsymbol{x}}\|_{1}$. Thus, the concentration inequality of Theorem 1 says that the estimate $\widehat{\boldsymbol{\theta}}$ of parameter ${\boldsymbol{\theta}}$ behaves as if it resulted from applying a separable proximal operator to observations in the Gaussian sequence model (1.1) at a particular noise variance $\tau^{2}$. Once we understand the dependence of $\tau,\lambda$ on the parameters of the problem (which [MM18] describes), Theorem 1 reduces the study of the LASSO in the linear model with Gaussian designs to the study of soft-thresholding in the scalar statistical model $y=\theta+\tau z$ with $\theta\sim\mu_{{\boldsymbol{\theta}}}$ and $z\sim\mathsf{N}(0,1)$ independent of $\theta$. For example, Theorem 1 implies that the realized $\ell_{2}$-loss of the LASSO estimate, $\frac{1}{p}\|\widehat{\boldsymbol{\theta}}-{\boldsymbol{\theta}}\|^{2}$, concentrates on the $\ell_{2}$-risk of soft-threshdolding in the scalar model, ${\mathbb{E}}_{\theta\sim\mu_{{\boldsymbol{\theta}}},z\sim\mathsf{N}(0,1)}[(\eta_{\mathsf{soft}}(\theta+\tau z;\lambda\xi)-\theta)^{2}]$.

In this paper, we show that a concentration inequality of the form (1.10) holds for any symmetric penalty $f_{p}$. In particular, the estimator (1.4) in the linear model (1.2) behaves as if it resulted from applying a certain separable proximal operator to observations in the Gaussian sequence model at a particular noise variance. One consequence of this result is that for a fixed empirical parameter distribution $\mu_{{\boldsymbol{\theta}}}$, aspect ratio $n/p$, noise variance $\sigma^{2}$, and symmetric penalty $f_{p}$, there exists a separable penalty which produces estimates in model (1.2) with nearly the same statistical properties.

1.3 Adaptive estimation

The preceding two sections suggest that in the models (1.1) and (1.2) and for a fixed empirical parameter distribution, non-separable symmetrically penalized least-squares behaves like separable symmetrically penalized least-squares with appropriately chosen penalty. What advantage, then, do non-separable penalties have for estimation?

We propose that one potential advantage of non-separability emerges when the empirical distribution of the parameters is unknown. Figure 1 suggests, and we show, that in the sequence model, non-separable symmetrically penalized least squares approximately applies to each coordinate an estimator which depends upon the underlying empirical parameter distribution. In this sense, symmetric proximal operators in the sequence model with unknown empirical distribution of the parameter automatically implement adaptive procedures. In particular, they select in a data-driven way a scalar estimator to (approximately) apply coordinate-wise. For any given penalty $f_{p}$, we characterize this adaptive procedure. Moreover, we provide a sufficient condition which given an adaptive procedure guarantees it is implemented by some symmetric, convex penalty.

Ideally, we could design a non-separable penalty which, for each underlying parameter distribution of interest, chose the “right” coordinate-wise estimator according to some criterion. While we do not explore the design of such penalties in this paper, we hope this work opens the door to principled approaches to designing non-separable M-estimators like the square-root LASSO and SLOPE [BCW11, BvdBS*+*15] which adapt to a pre-specified nuisance parameter.

1.4 Summary of contributions

To summarize and add to the preceding three sections, we list our primary contributions.

We prove a finite sample concentration inequality which quantifies in a precise sense the “asymptotic separability” of symmetric proximal operators in the Gaussian sequence model. This concentration inequality holds uniformly over all choices of penalty. This result is substantially more general and more precise than a similar result of [HL19]: more general because it applies to arbitrary symmetric penalties rather than only to penalties of the form (1.9); more precise because it holds in finite samples rather than asymptotically. 2. 2.

We prove a finite sample concentration inequality which –given a solution to a certain system of equations involving model and estimator parameters– characterizes in a precise sense the behavior of symmetrically penalized least squares in the linear model. The characterization involves comparison to a particular scalar estimation model. For the same reasons as above, this result is substantially more general and more precise than similar results in [HL19] which apply only to SLOPE and only asymptotically. Moreover, for each fixed empirical parameter distribution, aspect ratio $n/p$, noise variance $\sigma^{2}$, and symmetric penalty $f_{p}$, our results establish the same concentration inequality (with the same constants) for penalized least squares with a particular separable penalty. Thus, we reveal a near equivalence in behavior between the symmetrically and the separably penalized estimators (contingent on the existence of solutions to a certain system of equations). 3. 3.

A recent paper established a lower bound on the asymptotic squared error of symmetrically penalized least squares in the linear model with Gaussian design matrices with independent entries [CM19]. The arguments of that paper easily imply an analagous lower bound for separably penalized least squares. We show that the two lower bounds agree, which is significant because the lower bounds are expected to be generally tight. This and contribution 2 provide strong evidence that, in high dimensions, symmetrically penalized least squares cannot outperform separably penalized least squares in the linear model with Gaussian designs when the empirical parameter distribution is approximately known. 4. 4.

We argue that symmetric proximal operators in the sequence model with unknown empirical parameter distribution automatically implement adaptive procedures. We describe these adaptive procedures and provide a sufficient condition which given an adaptive procedure guarantees it is implemented by some symmetric, convex penalty. 5. 5.

We develop a theory of symmetric, convex penalties and proximal operators based on the tools of optimal transport theory. This theory is the basis of the contributions listed above. In the sequence model, this theory can generate results which generalize contribution 1 beyond the case of Gaussian noise. While the results we present hold with high-probability, the framework we develop can easily generate results which hold in other senses, like in expectation.

1.5 Related literature

Several recent works have developed precise characterizations of M-estimators in linear models with Gaussian designs. One approach, and the one we adopt, uses Gaussian comparison inequalities. This approach was first developed in [Sto13] and was developed further by [TOH15, TAH18]. Finite-sample concentration inequalities for the LASSO using this technique were developed by [MM18]. Sharp asymptotics for SLOPE in linear regression were provided by similar means in [HL19].

An alternative approach to the precise characterization of M-estimators in high-dimensional linear models uses Approximate Message Passing (AMP) algorithms [BM12, DM16, SC18]. The precise characterization of AMP algorithms via the so-called state evolution was first developed in [Bol14, BM11, JM13]. Recent work establishes the validity of the state evolution for such algorithms which use nonseparable non-linearities [BMN19]. This work was used in [CM19] to develop lower bounds on the asymptotic performance of symmetrically (and possibly nonseparably) penalized least squares.

A third approach to the precise characterization of M-estimators in the high-dimensional linear model uses leave-one-out techniques. This approach has been successfully applied to ridge regression and to schemes which penalize residuals using a non-quadratic but separable loss [EKBB*+*13, EK13].

The recent paper [HL19] identified the asymptotic separability of the SLOPE penalty, and our work is a natural extension of that paper.

The adaptive potential of M-estimation has been previously observed through specific examples. These include the square-root LASSO, which adapts to unknown noise-level by applying the loss $\|{\boldsymbol{y}}-{\boldsymbol{X}}{\boldsymbol{\theta}}\|/\sqrt{n}$ to the residuals [BCW11] and SLOPE, which adapts to unknown sparsity by applying a LASSO-like penalty which penalizes parameter estimates more or less strongly based on rank statistics [BvdBS*+*15, SC16].

Some of the results we develop regarding symmetric, convex functions are not entirely new. The papers [HW88, Day73] and the references therein develop several results regarding the structure of symmetric, convex functions in Banach spaces and their subdifferentials. We use the tools of optimal transport theory to the study of symmetric, convex penalties, which we believe is particularly fruitful in generating statistical insight into the behavior of M-estimation with symmetric penalties.

1.6 Notations

For a vector ${\boldsymbol{x}}\in\mathbb{R}^{p}$, we denote by $\mu_{{\boldsymbol{x}}}$ the empirical distribution of its coordinates; that is, $\mu_{{\boldsymbol{x}}}=\frac{1}{p}\sum_{j=1}^{p}\delta_{x_{j}}$. Similarly, for a random variable $X$, we denote by $\mu_{X}$ its law. We denote by $\bar{\mathbb{R}}=\mathbb{R}\cup\{\infty\}$ the real line completed at $\infty$. For two measures $\mu,\mu^{\prime}\in{\mathcal{P}}_{2}(\mathbb{R})$, we denote by $\Pi(\mu,\mu^{\prime})$ the collection of couplings between $\mu$ and $\mu^{\prime}$. That is, $\nu$ a probability measure on $\mathbb{R}^{2}$ is in $\Pi(\mu,\mu^{\prime})$ if its first marginal is $\mu$ and its second marginal is $\mu^{\prime}$. The set $\Pi(\mu,\mu^{\prime})$ is non-empty because it contains, in particular, the product meausure $\mu\otimes\mu^{\prime}$. It is well known that

[TABLE]

defines a metric $W_{2}(\mu,\mu^{\prime})$ on the space of probability measures with bounded second moment [Vil10, Definition 6.4]. We denote by ${\mathcal{P}}_{2}(\mathbb{R})$ the space of probability measures with finite second moment endowed with this metric. The space ${\mathcal{P}}_{2}(\mathbb{R})$ is refered to as the Wasserstein space of order 2 on $\mathbb{R}$. The optimal coupling between $\mu$ and $\mu^{\prime}$ is denoted $\pi_{\mathrm{opt}}(\mu,\mu^{\prime})$. For any $\mu\in{\mathcal{P}}_{2}(\mathbb{R})$ and $\tau\geq 0$, we denote by $\mu^{*\tau}=\mu*{\sf N}(0,\tau^{2})$, that is, the distribution of $\Theta+\tau Z$ when $\Theta\sim\mu,\,Z\sim{\sf N}(0,1)$ independent. For a vector ${\boldsymbol{x}}\in\mathbb{R}^{p}$, we denote $\|{\boldsymbol{x}}\|=\sqrt{\sum_{j=1}^{p}x_{j}^{2}}$ the standard Euclidean norm. We denote the space of square-integrable, Borel-measureable random variables on the unit interval with Lebesgue measure by $L_{2}(0,1)$. For $X,Y\in L_{2}(0,1)$, we denote $d_{2}(X,Y)=\sqrt{{\mathbb{E}}[(X-Y)^{2}]}$ the standard Hilbert-space metric, and $\|X\|_{2}=\sqrt{{\mathbb{E}}[X^{2}]}$ the standard Hilbert-space norm. The space ${\mathcal{P}}_{2}(\mathbb{R}^{k})$, the Wasserstein space of order 2 on $\mathbb{R}^{k}$, is defined similarly to ${\mathcal{P}}_{2}(\mathbb{R})$, replacing probability measures with finite second moment on $\mathbb{R}$ with those with finite second moment on $\mathbb{R}^{k}$ and replacing squared distance ${\mathbb{E}}[(X-X^{\prime})^{2}]$ with ${\mathbb{E}}[\|{\boldsymbol{X}}-{\boldsymbol{X}}^{\prime}\|^{2}]$.

1.7 Organization

In Section 2, we present and discuss our main results. In Section 3, we develop a theory which uses optimal transport to study symmetric convex functions. In Section 4, we prove our main results using the theory developed in Section 3. In fact, in Section 4 we prove slightly more general versions of the theorems which appear in Section 2. We defer these more general statements to Section 4 because their meaning and statistical relevance are less readily apparent. Nevertheless, they may serve as a more appropriate starting point for further extensions or applications to alternative models. Some technical details and additional simulations we defer to the appendices.

2 Main results

2.1 Limited advantages to non-separability in the Gaussian sequence model

In this section, we argue that estimators of the form (1.3) with symmetric, convex $f_{p}$ in the sequence model (1.1) behave as if they were defined using separable $f_{p}$. By (1.6), M-estimation with symmetric, separable penalties in the sequence model constructs an estimate by applying to each coordinate the same scalar estimator from the collection

[TABLE]

This collection does not contain all scalar functions or even all scalar, non-decrasing functions.

Fact 2.1**.**

The collection ${\mathcal{PR}}_{1}$ contains exactly those functions $\mathbb{R}\rightarrow\mathbb{R}$ which are non-decreasing and $1$-Lipschitz.

Fact 2.1 is proved in Appendix B.1. Our main result for estimation in the Gaussian sequence model states that all symmetric proximal operators construct an estimate by approximately applying to each coordinate the same scalar estimator from the collection ${\mathcal{PR}}_{1}$.

Theorem 2**.**

There exist (i) universal functions $\mathsf{c},\mathsf{C}:\mathbb{R}_{>0}\rightarrow\mathbb{R}_{>0}$, non-increasing and non-decreasing respectively, and (ii) for each $p$, a collection of mappings ${\mathfrak{A}}_{f_{p}}:{\mathcal{P}}_{2}(\mathbb{R})\rightarrow{\mathcal{PR}}_{1}$ indexed by lsc, proper, symmetric, convex functions $f_{p}:\mathbb{R}^{p}\rightarrow\bar{\mathbb{R}}$ such that the following is true.

For any $p$, ${\boldsymbol{\theta}}\in\mathbb{R}^{p}$, $\tau>0$, and $\epsilon\in(0,1/2]$, we have

[TABLE]

where $\mu^{*\tau}:=\mu*{\sf N}(0,\tau^{2})$, $C=\mathsf{C}(\mathsf{snr})$ and $c=\mathsf{c}(\mathsf{snr})$ with $\mathsf{snr}:=\frac{\|{\boldsymbol{\theta}}\|^{2}}{p\tau^{2}}$, the probability is over ${\boldsymbol{z}}\sim{\sf N}(0,I_{p})$, and it is understood that ${\mathfrak{A}}_{f_{p}}(\mu^{*\tau})$ is applied to ${\boldsymbol{\theta}}+\tau{\boldsymbol{z}}$ coordinate-wise. If $\tau=0$, we may replace the upper bound on the right-hand side by 0.

If $f_{p}({\boldsymbol{x}})$ is separable as in (1.5), we may take ${\mathfrak{A}}_{f_{p}}(\mu)=\mathsf{prox}[\rho]$ for all $\mu$.

Theorem 2 is proved in Section 4.1. A detailed description of how to construct such a collection of mappings ${\mathfrak{A}}_{\cdot}$ is deferred to Section 3, and in particular, Section 3.6. Importantly, the scalar estimator ${\mathfrak{A}}_{f_{p}}(\mu^{*\tau})$ appearing in Theorem 2 does not depend upon the realization of the noise ${\boldsymbol{z}}$. Rather, it depends only on the distribution $\mu^{*\tau}$, so that it can in principle be determined by the statistician in advance of any observations. Moreover, it can often be efficiently computed, as we briefly discuss in Section C.

We should think of the $\mu$ in Theorem 2 as a good approximation to the empirical distribution of the coordinates of ${\boldsymbol{\theta}}$. For example, we may take $\mu=\mu_{{\boldsymbol{\theta}}}$, in which case (2.2) simplifies to

[TABLE]

Keep in mind that (2.3) differs from (2.2) not only in the threshold which appears on the right-hand side of the inequality inside the probability, but also potentially in the scalar estimator: ${\mathfrak{A}}_{f_{p}}(\mu_{{\boldsymbol{\theta}}}^{*\tau})$ rather than ${\mathfrak{A}}_{f_{p}}(\mu^{*\tau})$.

We state the more general inequality (2.2), which applies to choices other than $\mu=\mu_{{\boldsymbol{\theta}}}$, because we may not always wish to apply Theorem 2 with this particular choice. For example, if we allow that ${\boldsymbol{\theta}}$ be random with coordinates drawn iid from $\mu$, then we may wish to take $\mu$ as the population rather than empiricial distribution of the cordinates of ${\boldsymbol{\theta}}$. With this choice, we may take any $M>0$ and view Eq. (2.2) as a bounded on the conditional probability conditioned on $\|{\boldsymbol{\theta}}\|^{2}/p\leq M$ with $C=\mathsf{C}(M/\tau^{2})$ and $c=\mathsf{c}(M/\tau^{2})$, with $W_{2}(\mu,\mu_{{\boldsymbol{\theta}}})$ now also random. Here we have used critically the monotonicity of $\mathsf{c},\mathsf{C}$. For $M$ larger than the second moment of $\mu$, the event $\|{\boldsymbol{\theta}}\|^{2}/p\leq M$ will hold with high probability. By quantifying this probability, the bound (2.2), which in this context holds only conditionally, can lead to an unconditional bound. We can often also control the probability that $W_{2}(\mu,\mu_{{\boldsymbol{\theta}}})$ is large (see e.g. [FG15]), so that with some work we can control the probability that $\frac{1}{p}\|\mathsf{prox}[f]({\boldsymbol{\theta}}+\tau{\boldsymbol{z}})-{\mathfrak{A}}_{f_{p}}(\mu^{*\tau})({\boldsymbol{\theta}}+\tau{\boldsymbol{z}})\|^{2}$ exceeds a parameter-independent threshold. Importantly, in this case the scalar estimator ${\mathfrak{A}}_{f_{p}}(\mu^{*\tau})$ does not depend upon the realization of ${\boldsymbol{\theta}}$ or ${\boldsymbol{z}}$. The distribution $\mu^{*\tau}=\mu*{\sf N}(0,\tau^{2})$ is interpreted as the population measurement distribution.

Theorem 2 says that for the purposes of estimation in the Gaussian sequence model (1.1) with $\mu_{{\boldsymbol{\theta}}}$ approximately known, non-separable, symmetric M-estimation behaves almost equivalently to separable, symmetric M-estimation with appropriately chosen convex penalty $\rho:\mathbb{R}\rightarrow\bar{\mathbb{R}}$. Importantly, because the functions $\mathsf{c},\mathsf{C}$ are universal, the rate of concentration we establish is uniform over choices of penalty $f_{p}$. That is, separable M-estimation approximates non-separable M-estimation uniformly well over such choices.

We remark that Theorem 2 follows from a more general theorem whose statement can be found in Section 4.1. For random ${\boldsymbol{\theta}}$, this more general theorem may yield tighter results than the approach outlined in the paragraph above. Moreover, this more general theorem is not specific to the model (1.1). From it, we can derive results analogous to Theorem 2 in different statistical models on $\mathbb{R}^{p}$. As we will see, the concentration we establish will always be uniform over the choice of $f_{p}$. In fact, it will only depend upon the rate of concentration of the empirical distribution of observations in Wasserstein space, a property of the statistical model and not the estimator we choose. Thus, Theorem 2 and its more general statement in Section 4.1 separate the analysis of the penalty, used to determine ${\mathfrak{A}}_{f_{p}}$, and the analysis of the statistical model, used to determine the rate of concentration.

2.2 No first-order asymptotic advantage to non-separability in the linear model

A recent paper [CM19] establishes an asymptotic lower-bound on the $\ell_{2}$-risk of the estimator (1.4) in a certain high-dimensional limit in which the empirical distribution of the coordinates of ${\boldsymbol{\theta}}$ appropriately converges. Here we prove that the lower bound established there over the collection of symmetric penalties agrees with the corresponding lower bound over the much smaller collection of separable penalties. Thus, we confirm a conjecture stated in a footnote and in Appendix Q of [CM19]. This equivalence is significant because these lower bounds are expected to be generally tight.

First we describe the lower-bound of [CM19].222The formulas differ slightly here because we adopt an slightly different convention of normalization. Fix prior $\mu\in{\mathcal{P}}_{2}(\mathbb{R})$. Consider a sequence $p\rightarrow\infty$. Let

[TABLE]

be the collection of all sequences of lsc, proper, symmetric, convex functions. Define the optimal per-coordinate $\ell_{2}$-risk of symmetric, penalized least squares in the Gaussian sequence model by

[TABLE]

where $\theta_{j}\stackrel{{\scriptstyle\mathrm{iid}}}{{\sim}}\mu$ and ${\boldsymbol{z}}\sim{\sf N}(0,I_{p})$ independent of ${\boldsymbol{\theta}}$. Let $\sigma^{2}\geq 0$. Define

[TABLE]

The lower-bound is given in the following proposition, which we copy from [CM19].

Theorem 3** (Theorem 1 of [CM19]).**

Consider a sequence of models (1.2) with $n,p\rightarrow\infty$, $n/p\rightarrow\delta\in(0,\infty)$. Assume that ${\boldsymbol{w}}$ is independent of ${\boldsymbol{X}}$ and almost surely

[TABLE]

(in particular, we consider both models in which ${\boldsymbol{\theta}},{\boldsymbol{w}}$ are random and models in which they are deterministic). If we adopt the convention that $\|\widehat{\boldsymbol{\theta}}-{\boldsymbol{\theta}}\|^{2}$ is infinite whenever the minimizing set in (1.4) is empty, then

[TABLE]

where a sequence of lsc, proper, symmetric, convex functions $\{f_{p}\}$ is in ${\mathcal{C}}_{\delta,\mu}$ if

[TABLE]

where in the expectation we take $\theta_{j}\stackrel{{\scriptstyle\mathrm{iid}}}{{\sim}}\mu$ and ${\boldsymbol{z}}\sim{\sf N}(0,I_{p})$ independent of ${\boldsymbol{\theta}}$.

Theorem 3 establishes a lower bound on the realized $\ell_{2}$-loss of symmetrically penalized least-squares asymptotically for sequences of penalties belonging to the collection ${\mathcal{C}}_{\delta,\mu}$. Using Fatou’s lemma, it is straightforward to extend the lower bound (2.8) to a lower-bound on the asymptotic risk of the estimator (1.4) (that is, where we take an expectation in (2.8). See [CM19, Lemma I.1]). The collection ${\mathcal{C}}_{\delta,\mu}$ is extensively discussed in [CM19], see for example Section 3 and Appendix C in that paper. It is argued there that the condition (2.9), though difficult to understand, is not very restrictive. The authors of [CM19] refer to sequences which satisfy (2.9) as sequences with $\delta$-bounded width, a terminology which we adopt as well.

Now we consider symmetric, separable penalties. Let ${\mathcal{F}}_{1}$ be the collection of all univariate, lsc, proper, convex functions. Define

[TABLE]

where $\theta\sim\mu$, $z\sim{\sf N}(0,1)$ independent, and the infimum is taken over all lsc, proper, convex functions. Define

[TABLE]

We claim that under the conditions of Theorem 3, we have that

[TABLE]

where the reader should have in mind that for each $p$ we make the same choice $f_{1}$, though this is not required. Indeed, it is straightforward (though perhaps tedious) to check that the proofs in [CM19] all go through if we instead consider estimation using separable penalties in the class ${\mathcal{C}}_{\delta,\mu}$ with the lower bound (2.12) (which includes, for example, all $f_{1}$ with unique minimizer [CM19, Claim 3.5]).

Our next result establishes that the right-hand sides of (2.8) and (2.12) agree.

Theorem 4**.**

Let $\mu\in{\mathcal{P}}_{2}(\mathbb{R})$ and $\tau\geq 0$. Then

[TABLE]

In particular, the lower-bound (2.8) on the optimal $\ell_{2}$-loss over sequences of symmetric penalties with $\delta$-bounded width agrees with the lower bound (2.12) over sequences of symmetric, separable penalties with $\delta$-bounded width.

Theorem 4 is proved in Section 4.2. Theorem 4 should not be surprising in light of Theorem 2. Indeed, $\mathsf{R_{symm}^{opt}}(\tau;\mu)$ and $\mathsf{R_{sep}^{opt}}(\tau;\mu)$ are defined in terms of the performance of M-estimation in the Gaussian sequence model. Theorem 4 suggests that asymptotically there is no advantage to non-separability in symmetrically penalized least squares for model (1.2). Confirming this claim requires establishing that the lower-bounds (2.8) and (2.12) are tight. While we believe this to be the case, we do not prove so here. In the following section we present further evidence of this fact. In particular, given a solution to a certain system of equations, we establish a finite-sample concentration inequality which characterizes the behavior of symmetrically penalized least-squares in the model (1.2) which also holds for a particular choice of separable, symmetric penalty.

Finally, we remark that Theorem 4 allows us to compute $\mathsf{R_{symm}^{opt}}(\tau;\mu)$ and the symmetric lower bound by computing $\mathsf{R_{sep}^{opt}}(\tau;\mu)$ and the separable lower bound. Because the quantities appearing in (2.10) involve only functions of a scalar random variables, they can be computed efficiently numerically (see [CM19, Appendix Q]), whereas the quantities in (2.5) may not be. The theoretical lower bounds presented in Table 1 and Figure 1 of [CM19] are computed via (2.10), and Theorem 3 rigorously justifies this computation.

2.3 Finite-sample behavior of symmetric M-estimators in the linear model

We now present a concentration inequality analogous to that in Theorem 2 except applied to the linear model (1.2). Our concentration inequality in the linear model uses a solution to a certain system of equations involving scalar estimators (and in particular, assumes such a solution exists).

Theorem 5**.**

There exist (i) universal functions $\mathsf{c_{1}},\mathsf{c_{2}},\mathsf{c_{3}},\mathsf{C}:\mathbb{R}_{>0}^{4}\rightarrow\mathbb{R}_{>0}$ and (ii) for each $p$, a collection of mappings ${\mathfrak{A}}_{f_{p}^{\prime}}:{\mathcal{P}}_{2}(\mathbb{R})\rightarrow{\mathcal{PR}}_{1}$ indexed by lsc, proper, symmetric, convex functions $f_{p}^{\prime}:\mathbb{R}^{p}\rightarrow\bar{\mathbb{R}}$ such that the following is true.

Fix any lsc, proper, symmetric, convex function $f_{p}:\mathbb{R}^{p}\rightarrow\bar{\mathbb{R}}$. Assume ${\boldsymbol{\theta}}\in\mathbb{R}^{p}$ with $\|{\boldsymbol{\theta}}\|^{2}/p\leq M$ for some $M$. Consider model (1.2) with $n/p=\delta$ and ${\boldsymbol{w}}$ independent of ${\boldsymbol{X}}$. Let $\widehat{\boldsymbol{\theta}}$ be any random vector satisfying (1.4) almost surely on the event that the minimizing set is non-empty. Consider random variables $\Theta^{*}\sim\mu_{{\boldsymbol{\theta}}}$ and $G^{*}\sim{\sf N}(0,1)$ independent and $\tau^{*},\sigma^{*}>0$. Let $Y^{*}=\Theta^{*}+\tau^{*}G^{*}$. Assume that $\tau^{*},\lambda^{*}$ solve

[TABLE]

where for any function $\eta:\mathbb{R}\rightarrow\mathbb{R}$ and random variable $X$, the expression $\eta\circ X$ denotes the random variable constructed by applying $\eta$ to the realized value of $X$.333We adopt this notation because we will frequently consider function $f:L_{2}(0,1)\rightarrow\bar{\mathbb{R}}$, and we do not wish to confuse a real-valued function of a random variables with the application of a real-valued function of a real number to the realized value of the random variable. Let $c_{j}=\mathsf{c_{j}}(M,\tau^{*},\sigma^{*},\delta)$, $j=1,2,3$ and $C=\mathsf{C}(M,\tau^{*},\sigma^{*},\delta)$. Then for all $0<\epsilon<c_{1}$,

[TABLE]

If $f_{p}^{\prime}({\boldsymbol{x}})$ is separable as in (1.5), we may take ${\mathfrak{A}}_{f_{p}^{\prime}}(\mu)=\mathsf{prox}[\rho]$ for all $\mu$.

Theorem 5 is proved in Section 4.3. The dependence of $\mathsf{c_{1}},\mathsf{c_{2}},\mathsf{c_{3}},\mathsf{C}$ on the parameters of the problem is careful tracked in its proof (and, in particular, in the part of the proof deferred to the Appendices). The collections \{{\mathfrak{A}}_{f_{p}}\mid f_{p}\text{ lsc, proper, convex on \mathbb{R}^{p}}\} for which Theorem 2 and Theorem 5 hold can be taken to be the same. We will describe these collections in Section 3.3. Theorem 5 supports the discussion following Theorem 4 that the equivalence between the symmetric and separable lower bounds is not an artefact of known proof techniques but rather is fundamental. Indeed, the constants $c_{1},c_{2},c_{3},C$ do not depend upon the choice of penalty $f_{p}$. In particular, the same constants apply to the separable penalty (1.5) which takes $\rho$ such that $\mathsf{prox}[\lambda^{*}\rho]={\mathfrak{A}}_{\lambda^{*}f_{p}}(\mu^{*\tau^{*}})$. Theorem 5 does not establish this equivalence completely because it requires assuming a solution to (2.15) exists, and it is difficult to argue that an M-estimator for which no solution exists could not perform better. In fact, much of the technical work in proving Theorem 3 goes into establishing the existence of a solution to a system like (2.14) [CM19]. Nevertheless, solutions to (2.14) should usually exist (for example, they always exist for the LASSO, see [MM18]), and when they do not, it often results from the estimator being severely ill-defined (for example, taking $f_{p}=0$ when $\delta<1$).

2.4 Advantages to non-separability: adaptive estimation in the sequence model

The mapping ${\mathfrak{A}}_{f_{p}}$ which appears in Theorems 2 and 5 has a natural adaptive interpretation. Empirical Bayes methods of estimation are based upon the insight that in certain high-dimensional statistical models the prior which generated the data can be consistenly estimated from the empirical measurement distribution, whence the Bayes estimator can be approximated and applied to estimate each parameter [Rob56, BG09, JZ09, Efr11].444Under certain interpretations, we need not view the underlying parameters as random. Rather, the prior can be viewed as a description of the fixed empirical distribution of the truth. Thus, Bayes performance can be approximately achieved without knowledge of the prior. We may view such full empirical Bayes methods as selecting a coordinate-wise estimator from the collection of all measurable estimators in a data-dependent way. We may also consider restricted empirical Bayes, in which the statistician uses the data to select an estimator from a restricted class rather than from the collection of all estimators. In these settings, for certain underlying priors the statisticians fail to consistently select the Bayes estimator. James-Stein estimation and procedures which adaptively choose a soft-thresholding parameter in sparse estimation are instances of the use of such techniques. See, for example, [EM73, DJ95, XKB12].

Theorems 2 and 5 indicate that symmetric penalties automatically implement exactly this type of program. For example, consider the sequence model (1.1) and proximal operators (1.3). For $f_{p}$ fixed, we should think of the mappings ${\mathfrak{A}}_{f_{p}}$ as a population adaptive mapping: it takes the true population measurement distribution (which is unknown) and chooses a scalar estimator to apply coordinate-wise based on that distribution. Theorem 2 says that symmetric proximal operators approximately implements the population adaptive mapping with errors bounded by (2.2). Some such error is inevitable. In finite samples, we cannot exactly infer the population measurement distribution, so we cannot exactly choose a scalar estimator to apply coordinate-wise based on ${\mathfrak{A}}_{f_{p}}$. Whether inequality (2.2) captures the correct or optimal error rate is not something we address. Conveniently, however, the error in inequality (2.2) is uniform over choices of symmetric penalty $f_{p}$.

The adaptive interpretation of M-estimators (1.4) in the linear model (1.2) is more complicated because the scalar estimator ${\mathfrak{A}}_{\lambda^{*}f_{p}}(\mu^{*\tau^{*}})$ depends upon $\lambda^{*}$, which is determined in a complicated manner via (2.14) (which, in turn, depends upon ${\mathfrak{A}}_{\lambda^{*}f_{p}}(\mu^{*\tau^{*}})$). Fixing the penalty $f_{p}$ and measurement rate $\delta$, the population adaptive mapping takes the parameter empirical distribution $\mu_{{\boldsymbol{\theta}}}$ and approximate noise level ${\sigma^{*}}^{2}$ and chooses a scalar estimator ${\mathfrak{A}}_{\lambda^{*}f_{p}}$ and noise level $\tau^{*}$ by finding a soultion to (2.14). Eq. (2.14) involves both the collection $\{{\mathfrak{A}}_{\lambda f_{p}}\}_{\lambda>0}$ and the parameters $\mu_{{\boldsymbol{\theta}}},\sigma^{*}$ to which it adapts. Theorem 5 says that symmetric least squares behaves approximately as if the scalar estimator ${\mathfrak{A}}_{\lambda^{*}f_{p}}$ selected by the population adaptive mapping were applied to measurements in the sequence model (1.1) at noise level $\tau^{*}$.

The adaptive potential of symmetric M-estimation should not be surprising. Recently, SLOPE was introduced to achieve FDR control and to adaptively achieve minimax rates of estimation over sparisty levels in both the Gaussian sequence model and the linear model [BvdBS*+*15, SC16, BGT18]. The current paper establishes in a much more general way the adaptive potential of symmetric M-estimation.

It is natural to ask which adaptive procedures can be implemented –exactly or approximately– by symmetrically penalized least squares in either the Gaussian sequence model or the linear model, and whether there is a principled design process which automates the discovery of an adaptive symmetric penalty for a particular task. In this paper, we characterize which adaptive procedures can be exactly implemented in the Gaussian sequence model and leave such a characterization in the linear model for future work.

More preciesly, consider a collection ${\mathcal{D}}\subset{\mathcal{P}}_{2}(\mathbb{R})$ of distributions with non-trivial Guassian component. That is, for each $\nu\in{\mathcal{D}}$, we may write $\nu=\mu*{\sf N}(0,\tau^{2})$ for some $\mu\in{\mathcal{P}}_{2}(\mathbb{R})$ and $\tau>0$. We should think of ${\mathcal{D}}$ as a collection population measurement distributions in a collection of sequence models of the form (1.1). Consider an ideal adaptive procedure defined by a mapping ${\mathfrak{A}}:{\mathcal{D}}\rightarrow{\mathcal{PR}}_{1}$. When does there exist a symmetric, convex $f_{p}$ such that we may take ${\mathfrak{A}}_{f_{p}}$ in Theorem 2 to agree with ${\mathfrak{A}}$ on ${\mathcal{D}}$? We provide an explicit characterization of the ${\mathfrak{A}}$ for which this is possible, which relies on the following notion.

Definition 2.2** (Joint cyclic monotonicity).**

A subset ${\mathfrak{R}}\subset{\mathcal{P}}_{2}(\mathbb{R}^{2})$ is jointly cyclically-monotone if for all finite $n$, all sets $\{\pi_{j}\}_{j=1}^{n}\subset{\mathfrak{R}}$, all random vectors $(X_{1},G_{1},\ldots,X_{n},G_{n})$ with $(X_{j},G_{j})\sim\pi_{j}$ for all $j$, and all permutations $\sigma:[n]\rightarrow[n]$, we have

[TABLE]

The implementability of an adaptive procedure via a symmetric proximal operator depends upon the joint cyclic monotonicity of a certain set.

Theorem 6**.**

There exist universal functions $\mathsf{c},\mathsf{C}:\mathbb{R}_{>0}\rightarrow\mathbb{R}_{>0}$, non-increasing and non-decreasing respectively, such that the following is true.

Consider ${\mathcal{D}}\subset{\mathcal{P}}_{2}(\mathbb{R})$ a collection of distributions, each with a non-trivial Gaussian component. Let ${\mathfrak{A}}:{\mathcal{D}}\rightarrow{\mathcal{PR}}_{1}$. If $\{\mu_{({\mathfrak{A}}(\mu_{Y})\circ Y,Y-{\mathfrak{A}}(\mu_{Y})\circ Y)}\mid\mu_{Y}\in{\mathcal{D}}\}$ is jointly cyclically monotone, then for each $p$ there exists an lsc, proper, symmetric, convex function $f_{p}:\mathbb{R}^{p}\rightarrow\bar{\mathbb{R}}$ such that for all ${\boldsymbol{\theta}}\in\mathbb{R}^{p}$, all $\mu\in{\mathcal{P}}_{2}(\mathbb{R})$ and $\tau>0$ with $\mu^{*\tau}\in{\mathcal{D}}$, and all $\epsilon\in(0,1/2]$, we have

[TABLE]

where $\mu^{*\tau}=\mu*{\sf N}(0,\tau^{2})$, $C=\mathsf{C}(\mathsf{snr})$ and $c=\mathsf{c}(\mathsf{snr})$ with $\mathsf{snr}=\frac{\|{\boldsymbol{\theta}}\|^{2}}{p\tau^{2}}$, the probability is over ${\boldsymbol{z}}\sim{\sf N}({\boldsymbol{0}},{\boldsymbol{I}}_{p})$, and it is understood that ${\mathfrak{A}}(\mu^{*\tau})$ is applied to ${\boldsymbol{\theta}}+\tau{\boldsymbol{z}}$ coordinate-wise.

Theorem 6 is proved in Section 4.4. The condition of Theorem 6 expresses the implementability of an adaptive procedure with a symmetric proximal operator through a condition that is intrinsic to the adaptive procedure itself. Unfortunately, the condition as stated is difficult to work with. We help make the statement more concrete with two examples.

Example 2.1** (Non-adaptive procedures).**

If ${\mathcal{D}}$ is a singleton, then the condition of Theorem 6 holds if and only if ${\mathfrak{A}}(\mu)$ is non-decreasing and 1-Lipschitz for the unique $\mu\in{\mathcal{D}}$. Indeed, in this case the separable penalty $f_{p}({\boldsymbol{x}})=\sum_{j=1}^{p}\rho(x_{j})$ where $\mathsf{prox}[\rho]={\mathfrak{A}}(\mu)$ implements the procedure ${\mathfrak{A}}$. This example is a non-adaptive example, so that its implementability by separable, symmetric M-estimation in unsurprising. If $\mu=\mu^{\prime}*{\sf N}(0,\tau^{2})$, we can implement the Bayes estimator ${\mathfrak{A}}(\mu)(y)={\mathbb{E}}[\theta|\theta+\tau z=y]$, where $\theta\sim\mu^{\prime}$ and $z\sim{\sf N}(0,1)$ independent of $\theta$, if and only if the Bayes estimator is 1-Lipschitz (it is non-decreasing automatically).

Example 2.2** (Full-empirical Bayes).**

Fix $\tau>0$ and let ${\mathcal{D}}=\{\mu^{*\tau}\mid\mu\in{\mathcal{P}}_{2}(\mathbb{R})\}$. Let ${\mathfrak{A}}(\mu^{*\tau})(y)={\mathbb{E}}[\theta|\theta+\tau z=y]$, where on the right-hand side $\theta\sim\mu^{\prime}$ and $z\sim{\sf N}(0,1)$ independent of $\theta$. The condition of Theorem 6 fails for this ${\mathfrak{A}}$. Indeed, there exist $\mu$ such that the Bayes estimator is not 1-Lipschitz, and we cannot construct $f_{p}$ such that ${\mathfrak{A}}_{f_{p}}(\mu^{*\tau})$ is the Bayes estimator at such $\mu$ by the preceding example.

The reader may wonder why adaptive procedures which can be implemented via symmetric M-estimation deserve special attention. We suggest several reasons. First, convex M-estimators should typically be easy to compute. Thus, we may expect that identifying adaptive procedures implemented by convex M-estimation will also generate procedures which are computationally feasible. Second, M-estimators designed for adaptation in one model may continue to exhibit appealing adaptive qualities in alternative models. For example, an important paper of Abramovich et al. [ABDJ06] demonstrated an intriguing connection between adaptive estimation and FDR control in the Gaussian sequence model: by only estimating those means selected by Benjamini-Hochberg at sufficiently low target FDR, one could achieve asymptotic minimaxity in estimation across sparsity levels with respect to several losses. Unfortunately, it is not obvious how to generalize their procedure to linear models. In contrast, SLOPE, which achieves FDR control and adaptive minimaxity in the sequence model, has an obvious generalization to the linear model (1.2). Thus, the identification of a penalty which behaves well in one model may more easily generate candidate procedures in alternative models that we may hope retain good properties. Indeed, Theorem 5 may serve as a starting point for understanding the use of a particular penalty in a linear model with Gaussian designs, but the qualitative behavior identified by such an analysis may generalize to less restrictive design assumptions. Third, the concentration inequalities we have established in Theorems 2 and 5 hold for any choice of convex $f_{p}$. A major challenge the statistician faces in designing adaptive procedures is controlling selection bias. Somehow, the convexity of the estimators controls selection bias in a manner which is uniform over choices of $f_{p}$. Theorems 2 and 5 permit the automatic control of bias arising from adaptivity without requiring a separate analysis for each penalty.

We conclude this section by providing several open questions which we believe may prove fruitful. First, which natural adaptive procedures beyond those of examples 2.1 and 2.2 do or do not satisfy the condition of Theorem 6? Second, is there a simpler characterization of the implementability of an adaptive procedure than that found in Theorem 6? Perhaps a weaker sufficient condition than that in Theorem 6 exists which guarantees its the result while being more interpretable and still widely applicable. Third, is there a notion of “approximate implementability” which captures when there exists an $f_{p}$ such that (2.2) holds with an ${\mathfrak{A}}^{\prime}\approx{\mathfrak{A}}$ in an appropriate sense? Insisting on exact implementability of a pre-specified procedure may be too restrictive and conceal the existence of high-quality adaptive penalties. Fourth, given a certain adaptive goal (adapting to sparsity in the Gaussian sequence model, for example), is there a principled design process by which we might automate the discovery of adaptive symmetric penalties implementing –exactly or approximately– a particular adaptive procedure or achieving certain minimax rates adaptive to a certain structural parameter? Finally, can we prove a theorem analagous to Theorem 6 for the linear model (1.2)?

3 Symmetric functions and optimal transport

The main strategy towards establishing the results in Section 2 is to view lsc, proper, symmetric, convex function on $\mathbb{R}^{p}$ as the restrictions of lsc, proper, symmetric, convex function on $L_{2}(0,1)$. Such a viewpoint has been developed in [HW88, Day73], for example. We develop this viewpoint from a different perspective by drawing on the tools of optimal transport theory, which we believe is particularly natural in statistical applications.

3.1 Two function spaces and their equivalence

We consider $\bar{\mathbb{R}}$-valued functions defined on $L_{2}(0,1)$ and on ${\mathcal{P}}_{2}(\mathbb{R})$. Functions defined on $L_{2}(0,1)$ will be denoted by standard font $f,g,$ etc., and those defined on ${\mathcal{P}}_{2}(\mathbb{R})$ will be denoted by fraktur font ${\mathfrak{f}},{\mathfrak{g}},$ etc.

Definition 3.1** (Symmetric functions).**

A function $f:L_{2}(0,1)\rightarrow\bar{\mathbb{R}}$ is symmetric if it is constant on the equivalence classes defined by the equivalence relations

[TABLE]

Equivalently, $f$ is symmetric if and only if there exists a function ${\mathfrak{f}}:{\mathcal{P}}_{2}(\mathbb{R})\rightarrow\bar{\mathbb{R}}$

[TABLE]

The structure of a symmetric function $f$ is reflected in the structure of the function ${\mathfrak{f}}$. In particular,

Proposition 3.2**.**

Consider a symmetric function $f:L_{2}(0,1)\rightarrow\bar{\mathbb{R}}$. Then, there exists a unique function ${\mathfrak{f}}:{\mathcal{P}}_{2}(\mathbb{R})\rightarrow\bar{\mathbb{R}}$ satisfying (3.2). Moreover,

1. (a)

$f$* is proper (i.e. not everywhere infinite) if and only if ${\mathfrak{f}}$ is proper.* 2. (b)

$f$* is lower semi-continuous if and only if ${\mathfrak{f}}$ is lower semi-continuous.* 3. (c)

$f$* is convex if and only if for all $\mu,\mu^{\prime}\in{\mathcal{P}}_{2}(\mathbb{R})$, any $\pi\in\Pi(\mu,\mu^{\prime})$, and any $\alpha\in[0,1]$, we have*

[TABLE]

where $\alpha\mu\oplus_{\pi}(1-\alpha)\mu^{\prime}\in{\mathcal{P}}_{2}(\mathbb{R})$ is defined a follows: construct $(X,X^{\prime})\sim\pi$ (see Lemma 3.4 below) and define $\alpha\mu\oplus_{\pi}(1-\alpha)\mu^{\prime}:=\mu_{\alpha X+(1-\alpha)X^{\prime}}$.

Throughout the paper, we will always use the topology on ${\mathcal{P}}_{2}(\mathbb{R})$ induced by the Wasserstein metric $W_{2}$. Proposition 3.2 justifies the following definition.

Definition 3.3** (Convexity on Wasserstein Space).**

A function ${\mathfrak{f}}:{\mathcal{P}}_{2}(\mathbb{R})\rightarrow\bar{\mathbb{R}}$ is convex if for any $\mu,\mu^{\prime}\in{\mathcal{P}}_{2}(\mathbb{R})$, coupling $\pi\in\Pi(\mu,\mu^{\prime})$, and $\alpha\in[0,1]$, Eq. (3.3) holds.

The proof of Proposition 3.2 relies on the following key embedding lemma, which will also be used repeatedly in the following sections. It allows us to realize multivariate probability distributions as the joint distributions of collections of random variables in $L_{2}(0,1)$.

Lemma 3.4**.**

We have the following.

1. (a)

For any $k\geq 1$ and $\pi\in{\mathcal{P}}_{2}(\mathbb{R}^{k})$, there exists $X_{1},X_{2},\ldots,X_{k}\in L_{2}(0,1)$ with $(X_{1},X_{2},\ldots,X_{k})\sim\pi$. 2. (b)

For any $\mu,\mu_{1},\mu_{2},\ldots\in{\mathcal{P}}_{2}(\mathbb{R})$ and $\pi_{p}\in\Pi(\mu,\mu_{p})$ for all $p$, there exists $X,X_{1},X_{2},\ldots\in L_{2}(0,1)$ such that $(X,X_{p})\sim\pi_{p}$ for all $p$. 3. (c)

For any sequence $\{\mu_{p}\}\subset{\mathcal{P}}_{2}(\mathbb{R})$ with $\mu_{p}\stackrel{{\scriptstyle\mathrm{W}}}{{\rightarrow}}\mu\in{\mathcal{P}}_{2}(\mathbb{R})$, there exists $X_{p},X\in L_{2}(0,1)$ with $\mu_{X_{p}}=\mu_{p}$ for all $p$, $\mu_{X}=\mu$, and $X_{p}\stackrel{{\scriptstyle L_{2}}}{{\rightarrow}}X$.

Such embedding results are common [Kal02]. We provide a proof in Appendix B.2 for the reader’s convenience. We can now prove Proposition 3.2.

Proof of Proposition 3.2.

Uniqueness holds because by Lemma 3.4.(a), for all $\mu\in{\mathcal{P}}_{2}(\mathbb{R})$, there exists a random variable $X\in L_{2}(0,1)$ with $\mu_{X}=\mu$. Thus, ${\mathfrak{f}}(\mu)$ is dicated by $f(X)$, and vice-versa.

1. (a)

(Proper) $f$ is proper if and only if for some $X\in L_{2}(0,1)$ we have $f(X)<\infty$, which by the embedding Lemma 3.4.(a) occurs exactly when for some $\mu\in{\mathcal{P}}_{2}(\mathbb{R})$ we have ${\mathfrak{f}}(\mu)<\infty$. 2. (b)

(Lower semi-continuous) Assume $f$ is lsc. Consider $\mu_{p}\stackrel{{\scriptstyle\mathrm{W}}}{{\rightarrow}}\mu$. Taking $X,X_{p}$ as in Lemma 3.4.(b), we have $\liminf_{p\rightarrow\infty}{\mathfrak{f}}(\mu_{p})=\liminf_{p\rightarrow\infty}f(X_{p})\geq f(X)={\mathfrak{f}}(\mu)$, whence ${\mathfrak{f}}$ is lsc. Conversely, assume ${\mathfrak{f}}$ is lsc. Consider $X_{p}\stackrel{{\scriptstyle L_{2}}}{{\rightarrow}}X$. By definition, we have $W_{2}(\mu_{X_{p}},\mu_{X})\leq d_{2}(X_{p},X)$, whence $\mu_{X_{p}}\stackrel{{\scriptstyle\mathrm{W}}}{{\rightarrow}}\mu_{X}$. Thus, $\liminf_{p\rightarrow\infty}f(X_{p})=\liminf_{p\rightarrow\infty}{\mathfrak{f}}(\mu_{X_{p}})\geq{\mathfrak{f}}(\mu_{X})=f(X)$, whence $f$ is lsc. 3. (c)

(Convex) Assume $f$ is convex. Consider $\mu,\mu^{\prime},\pi,\alpha$ as in the statement of the proposition. Taking $X,X^{\prime}$ as in Lemma 3.4.(a), we have ${\mathfrak{f}}(\alpha\mu\oplus_{\pi}(1-\alpha)\mu^{\prime})={\mathfrak{f}}(\mu_{\alpha X+(1-\alpha)X^{\prime}})=f(\alpha X+(1-\alpha)X^{\prime})\leq\alpha f(X)+(1-\alpha)f(X^{\prime})=\alpha{\mathfrak{f}}(\mu_{X})+(1-\alpha){\mathfrak{f}}(\mu_{X^{\prime}})$, whence we conclude (3.3). The converse follows via the same logic but by first taking arbitrary $X,X^{\prime}\in L_{2}(0,1)$, and then considering $\mu_{X},\mu_{X^{\prime}}$, and $\pi=\mu_{(X,X^{\prime})}$.

The proof is complete. ∎

Denote by ${\mathcal{F}}$ to the space of lsc, proper, symmetric, and convex functions on $L_{2}(0,1)$ and ${\mathfrak{F}}$ the space of lsc, proper, convex functions on ${\mathcal{P}}_{2}(\mathbb{R})$. Proposition 3.2 establishes a bijection between ${\mathcal{F}}$ and ${\mathfrak{F}}$ via (3.2).

3.2 Proximal operators on Wasserstein space

We define the proximal operator of $f\in{\mathcal{F}}$ as

[TABLE]

Because the objective in (3.4) is strongly convex and proper, its minimizer exists and is unique, so that $\mathsf{prox}[f]$ is well-defined. From the symmetry of $f$, one might naturally expect that the joint distribution of $(Y,\mathsf{prox}[f](Y))$ depends only on the distribution of $Y$, so that the proximal operator “inherits” the symmetry of $f$. While this intuition ends up being correct, its proof is not immediate. The primary difficulty is indicated by the following counter-example.

Example 3.1**.**

There exist $U_{1},U_{2}\in L_{2}(0,1)$ such that $U_{1},U_{2}\stackrel{{\scriptstyle\mathrm{iid}}}{{\sim}}\mathsf{Unif}(0,1)$. This follows from Lemma 3.4.(a). Nevertheless, there exists $U\in L_{2}(0,1)$ with $U\sim\mathsf{Unif}(0,1)$ such that $U$ is only independent of constant random variables. Indeed, if we take $U(\omega)=\omega$ for $\omega\in(0,1)$, then the sigma-algebra generated by $U$ is the Borel $\sigma$-algebra on $(0,1)$, which is only independent of the trivial $\sigma$-algebra $\{\emptyset,(0,1)\}$.555This example is related to the potential non-invertibility of measure-preserving maps on measure spaces, and previous authors have observed the challenge this poses in studying symmetric functions in infinite dimensional spaces. See, e.g. [HW88, pg. 463].

Example 3.1 indicates that the geometry of a random variable in $L_{2}(0,1)$ in relation to the rest of the space does not depend only upon its distribution. In particular, imagine that for a particular $Y\in L_{2}(0,1)$, the joint distribution of $(Y,\mathsf{prox}[f](Y))$ is $\pi$. Example 3.1 shows that, a priori, it could be the case that for some $\tilde{Y}\stackrel{{\scriptstyle\mathrm{d}}}{{=}}Y$, there exists no $X\in L_{2}(0,1)$ for which $(Y,X)\sim\pi$, in which case $(\tilde{Y},\mathsf{prox}[f](\tilde{Y}))$ could not have distribution $\pi$.

This scenario does not occur, and the reason is that the joint distribution between $Y$ and $\mathsf{prox}[f](Y)$ satisfies certain structural properties induced by the proximal minimization. We identify these structural properties by first studying proximal operators on Wasserstein space and later “lifting” results about such proximal operators to $L_{2}(0,1)$. We define the proximal operator of a function ${\mathfrak{f}}\in{\mathfrak{F}}$ as

[TABLE]

In Appendix B.3, we show that when ${\mathfrak{f}}$ is lsc, proper, and convex, the minimum on the right-hand side of (3.5) exists and is unique. Unlike $\mathsf{prox}[f]$, the proximal operator on Wasserstein space is automatically symmetric: by definition it depends only on the distribution $\mu$. This is one of the main benefits of developing first the theory for proximal operators on Wasserstein space before “lifting” it to $L_{2}(0,1)$.

An important object is $\pi_{\mathrm{opt}}(\mu,\mathsf{prox}[{\mathfrak{f}}](\mu))$, the optimal coupling between $\mu$ and the $\nu$ which minimizes the objective in (3.5). For any $\pi\in\mathbb{R}^{2}$, we denote by $\mathsf{spt}(\pi)$ the support of $\pi$, defined to be the intersection of all closed sets with measure 1 according to $\pi$. In particular, $\mathsf{spt}(\pi)$ is a closed set with measure 1 and is the smallest such set. A standard fact from optimal transport theory is that $\pi$ is the optimal coupling between its marginals if and only if $\mathsf{spt}(\pi)\subset\mathbb{R}^{2}$ satisfies the following property [Vil10, Theorem 5.10.(ii)].

Definition 3.5** (Cyclic monotonicity on $\mathbb{R}\times\mathbb{R}$).**

A set $\Gamma\subset\mathbb{R}\times\mathbb{R}$ is said to be cyclically monotone if for every $k\in{\mathbb{Z}}_{>0}$, every permutation $\sigma:[k]\rightarrow[k]$, and every finite family of points $(x_{1},y_{1}),\ldots,(x_{k},y_{k})\in\Gamma$ we have

[TABLE]

Equivalently,

[TABLE]

Equivalently, $\{x_{j}\}$ and $\{y_{j}\}$ have the same ordering. That is, there are no $i,j$ such that $x_{i}<x_{j}$ and $y_{i}>y_{j}$.

By [Vil10, Theorem 5.10.(ii)], $\mathsf{spt}(\pi_{\mathrm{opt}}(\mu,\mathsf{prox}[{\mathfrak{f}}](\mu)))$ is cyclically monotone. The proximal minimization (3.5) imposes the following additional structure.

Proposition 3.6**.**

Fix $\mu\in{\mathcal{P}}_{2}(\mathbb{R})$. Let $\pi_{\mathrm{res}}=\mu_{(Y-B^{*},B^{*})}$ where $(Y,B^{*})\sim\pi_{\mathrm{opt}}(\mu,\mathsf{prox}[{\mathfrak{f}}](\mu))$. Then $\pi_{\mathrm{res}}$ is the optimal coupling between $\mu_{Y-B^{*}}$ and $\mu_{B^{*}}$. In particular, both $\mathsf{spt}(\pi_{\mathrm{opt}})$ and $\mathsf{spt}(\pi_{\mathrm{res}})$ are cyclically monotone.

Proof of Proposition 3.6.

By [Vil10, Theorem 5.10.(ii)], the cyclic monotonicity of the given sets holds once we establish the optimality of the coupling $\pi_{\mathrm{res}}$.

Let $\rho_{\mathrm{opt}}$ be the optimal coupling between $\mu_{Y-B^{*}}$ and $\mu_{B^{*}}$. By Lemma 3.4.(b), we can construct $(R,B,B^{\prime})$ random variables on $L_{2}(0,1)$ such that $(R,B)\sim\pi_{\mathrm{res}}$ and $(R,B^{\prime})\sim\rho_{\mathrm{opt}}$. Define $Y^{\prime}=B+R$. Then $(Y^{\prime},B)\sim\mu_{(Y,B^{*})}=\pi_{\mathrm{opt}}$. For all $\alpha\in(0,1)$, we have

[TABLE]

where in the first equality we use $\mu_{B}=\mu_{B^{\prime}}$, in the first inequality we use (3.3) and the definition of the Wasserstein distance, in the the second we use (3.5), and in the final equality we use the optimality of $\pi_{\mathrm{opt}}$. Taking $\alpha\downarrow 0$ gives ${\mathbb{E}}[R(B^{\prime}-B)]={\mathbb{E}}[(Y^{\prime}-B)(B^{\prime}-B)]\leq 0$, whence ${\mathbb{E}}[(R-B)^{2}]={\mathbb{E}}[(R-B^{\prime})^{2}]+2{\mathbb{E}}[R(B^{\prime}-B)]\leq{\mathbb{E}}[(R-B^{\prime})^{2}]$, where the first equality uses ${\mathbb{E}}[B^{2}]={\mathbb{E}}[{B^{\prime}}^{2}]$. That is, $\pi_{\mathrm{res}}$ is the optimal coupling between $\mu_{Y-B^{*}}$ and $\mu_{B^{*}}$, as desired. ∎

3.3 Effective scalar estimators

One important consequence of Proposition 3.6 is that the optimal coupling that solves the proximal minimization (3.5) is implemented by a deterministic map. In later sections, we will see that this deterministic map, for a particular choice of ${\mathfrak{f}}$, is the ${\mathfrak{A}}_{f_{p}}$ which appears in Theorems 2, 5, and 6.

Proposition 3.7** (Effective scalar estimators).**

Let ${\mathfrak{f}}\in{\mathfrak{F}}$. Fix $\mu\in L_{2}(0,1)$. Then there exists $\eta\in{\mathcal{PR}}_{1}$ such that if $Y\sim\mu$, then $(Y,\eta\circ Y)\sim\pi_{\mathrm{opt}}(\mu,\mathsf{prox}[{\mathfrak{f}}](\mu))$.

Note that in the theory of optimal transport, there exist many pairs of probability distributions whose optimal coupling is non-deterministic (see, e.g. [Vil10, pg. 6]); that is, there exist no functions $\eta$ for which the preceding proposition holds. Moreover, when such functions exists, they may –by necessity– be discontinuous, and if continuous, need not be 1-Lipschitz. Thus, the optimization (3.5) imposes substantial additional structure.

Proposition 3.7 does not state that the $\eta$ which implements the optimal coupling is unique. Indeed, if $\mu$ does not have full support, it will in general not be unique because it will typically not be determined outside of the support. A central object will be mappings of the following type.

Definition 3.8** (Effective scalar representation).**

A mapping ${\mathfrak{A}}:{\mathcal{P}}_{2}(\mathbb{R})\rightarrow{\mathcal{PR}}_{1}$ is an effective scalar representation of ${\mathfrak{f}}\in{\mathfrak{F}}$ if for each $\mu\in L_{2}(0,1)$ we have

[TABLE]

If $f\in{\mathcal{F}}$ is related to ${\mathfrak{f}}$ by (3.2), then we say that ${\mathfrak{A}}$ is an effective scalar representation of $f$.

For all ${\mathfrak{f}}\in{\mathfrak{F}}$, Proposition 3.7 guarantees the existence of an effective scalar representation of ${\mathfrak{f}}$.

Proof of Proposition 3.7.

Define $\pi_{\mathrm{res}}$ as in Proposition 3.6. Because $(y,x)\mapsto(y-x,x)$ is a homeomorphism, we have

[TABLE]

By the cyclic monotonicity of $\mathsf{spt}(\pi_{\mathrm{res}})$ and (3.9), we also have that there exist no $(y_{0},x_{0}),(y_{1},x_{1})\in\mathsf{spt}(\pi_{\mathrm{opt}})$ such that $x_{0}<x_{1}$ and $y_{0}-x_{0}>y_{1}-x_{1}$. Thus, if $x_{0}<x_{1}$, then $y_{0}\leq y_{1}+x_{0}-x_{1}<y_{1}$. In particular, for each $y$ for which there exists $x$ with $(y,x)\in\mathsf{spt}(\pi_{\mathrm{opt}})$, that $x$ is unique, so we may define $\eta(y)=x$ for such $y$ and get

[TABLE]

By the cyclic monotonicity of $\mathsf{spt}(\pi_{\mathrm{opt}})$, there exists no $(y_{0},x_{0}),(y_{1},x_{1})\in\mathsf{spt}(\pi_{\mathrm{opt}})$ such that $y_{0}<y_{1}$ and $x_{0}>x_{1}$, whence $\eta$ is non-decreasing on its domain. Further, by the cyclic monotonicity of $\mathsf{spt}(\pi_{\mathrm{res}})$, there exists no $(y_{0},x_{0}),(y_{1},x_{1})\in\mathsf{spt}(\pi_{\mathrm{opt}})$ such that $y_{0}<y_{1}$ and $x_{1}-x_{0}>y_{1}-y_{0}$. Thus, $\eta$ is 1-Lipschitz on its domain. We may extend $\eta$ to a non-decreasing, 1-Lipschitz function on all of $\mathbb{R}$ by the Lipschitz extension theorem (see, e.g. [EG15]). Eq. (3.10) still holds for the extended $\eta$.666This is the only place where non-uniqueness occurs. Observe, non-uniqueness occurs only if $\mu$ does not have full support. Now, for any $Y\sim\mu$, we have $(Y,\eta\circ Y)\sim\pi_{\mathrm{opt}}$, as desired. ∎

3.4 Proximal operators on Hilbert space

We are now ready to establish the symmetry of proximal operators on $L_{2}(0,1)$, defined in (3.4). In addition, the following proposition gives us a convenient representation of these proximal operators.

Proposition 3.9**.**

Consider any lsc, proper, convex $f$.

1. (a)

$\mathsf{prox}[f]$* is 1-Lipschitz.* 2. (b)

Assume $f$ is also symmetric (i.e. $f\in{\mathcal{F}}$). A mapping ${\mathfrak{A}}:{\mathcal{P}}_{2}(\mathbb{R})\rightarrow{\mathcal{PR}}_{1}$ is an effective scalar representation of $f$ (see Definition 3.8) if and only if for all $Y\in L_{2}(0,1)$, we have

[TABLE]

In particular, for all $Y\in L_{2}(0,1)$, we have $\mathsf{prox}[f](Y)$ is $\sigma(Y)$-measurable.

Note that the right-hand side of (3.11) yields the symmetry of $\mathsf{prox}[f]$ we have promised. Indeed, the function ${\mathfrak{A}}(\mu_{Y})\in{\mathcal{PR}}_{1}$ depends on $Y$ only via its distribution. Then, the distribution of $(Y,{\mathfrak{A}}(\mu_{Y})\circ Y)$ is the push-forward of the measure $\mu_{Y}$ through the measurable (in fact, continuous) mapping $y\mapsto(y,{\mathfrak{A}}(\mu_{Y})(y))$, which also depends only on $\mu_{Y}$.

Proof of Proposition 3.9.

Part (a) is standard [PB13].

Now part (b). First assume ${\mathfrak{A}}$ is an effective scalar representation of $f$, and let ${\mathfrak{f}}\in{\mathfrak{F}}$ be related to $f$ via (3.2). For all $X\in L_{2}(0,1)$, we have

[TABLE]

where the first inequality follows from the definition of the Wasserstein metric and (3.2), the second inequality follows from (3.5), and the final equality from (3.8). Because the minimizer of (3.4) is unique, we have ${\mathfrak{A}}(\mu_{Y})\circ Y=\mathsf{prox}[f](Y)$. ∎

3.5 Subdifferentials of symmetric, convex functions

We now study the subdifferential relations of a function $f\in{\mathcal{F}}$ via the same strategy employed above to study their proximal operators. In particular, we study a suitably defined subdifferential of a function ${\mathfrak{f}}\in{\mathfrak{F}}$ and later “lift” the results to functions $f\in{\mathcal{F}}$. The motivation for pursuing this two-step strategy is, as above, to eliminate potential asymmetries arising from difficulties like that in Example 3.1.

Recall the subderivative of a convex function $f$ evaluated at $X$, denoted by $\partial f(X)$, is defined by

[TABLE]

The subdifferential $\partial f\subset L_{2}(0,1)\times L_{2}(0,1)$ is the relation defined by

[TABLE]

First, we recall what is known about such relations without the assumption of symmetry.

Definition 3.10** (Cyclic monotonicity on $L_{2}(0,1)\times L_{2}(0,1)$).**

A relation ${\mathcal{R}}\subset L_{2}(0,1)\times L_{2}(0,1)$ is cyclically-monotone if for every finite set $\{(X_{j},G_{j})\}_{j=1}^{n}\subset{\mathcal{R}}$ and every permutation $\sigma:[n]\rightarrow[n]$, we have

[TABLE]

A relation is maximally cyclically-monotone if it is cyclically monotone and not a proper subset of another cyclically monotone relation.

There is no clash of terminology between Definitions 3.5 and 3.10. Indeed, cyclic monotonicity can be defined for subsets of $H\times H$ where $H$ is any Hilbert space. In (3.7) the Hilbert space is $\mathbb{R}$, and in (3.14) the Hilbert space is $L_{2}(0,1)$.

In Theorem 1, the remark following Corollary 2, and Theorem 3 of [Roc66], Rockafellar establishes that (i) the subdifferential of an lsc, proper, convex function $f$ is maximally cyclically-monotone, (ii) conversely, any maximally cyclically montone relation is the subdifferential of an lsc, proper, convex function which is unique up to an additive constant, and (iii) every cyclically monotone relation is contained in the subdifferential of some lsc, proper, convex function. Our objective is to make similar statements for the case where $f$ is also symmetric. First, we observe that symmetry enables a more compact represention of the subdifferential relation.

Proposition 3.11**.**

If $f\in{\mathcal{F}}$, than $\partial f$ is symmetric in the sense that membership of $(X,G)$ in $\partial f$ is determined by the joint distribution $\mu_{(X,G)}$. That is, there exists a subset of ${\mathcal{P}}_{2}(\mathbb{R}^{2})$, which we will denote by ${\mathfrak{D}}f$, such that the following are equivalent

[TABLE]

Proof of Proposition 3.11.

By [BC11, Proposition 12.26], we have that $(X,G)\in\partial f$ if and only if $X=\mathsf{prox}[f](X+G)$. But $\mathsf{prox}[f](X+G)={\mathfrak{A}}_{{\mathfrak{f}}}(\mu_{X+G})\circ(X+G)$. Thus, $(X,G)\in\partial f$ if and only if $X={\mathfrak{A}}_{{\mathfrak{f}}}(\mu_{X+G})\circ(X+G)$. The latter condition only depends upon the joint-distribution $\mu_{(X,G)}$. ∎

We call the set ${\mathfrak{D}}f$ of Proposition 3.11 the Wasserstein subdifferential of the function $f$. For any ${\mathfrak{f}}\in{\mathfrak{F}}$, we define

[TABLE]

where ${\mathfrak{f}}$ is the unique element of ${\mathfrak{F}}$ identified with $f$ via (3.2). Unsurprisingly, ${\mathfrak{D}}{\mathfrak{f}}$ has an equivalent definition intrinsic to the function ${\mathfrak{f}}$.

Proposition 3.12**.**

For any ${\mathfrak{f}}\in{\mathfrak{F}}$, we have $\pi\in{\mathfrak{D}}{\mathfrak{f}}$ if and only if for all $(X,G,X^{\prime})$ with $(X,G)\sim\pi$ and $X^{\prime}\in L_{2}(0,1)$

[TABLE]

Proof of Proposition 3.12.

First, $\Rightarrow$. Let $f$ be related to ${\mathfrak{f}}$ via (3.2). Then, by Proposition 3.11, $(X,G)\sim\pi$ implies $G\in\partial f(X)$, whence $f(X^{\prime})\geq f(X)+{\mathbb{E}}[G(X^{\prime}-X)]$ by (3.12). By (3.2), this is equivalent to (3.17). Now, $\Leftarrow$. Assume (3.17) holds for all $(X,G,X^{\prime})$ of the specified form. By the coupling lemma (Lemma 3.4.(a)), we may construct at least one $(X,G)\sim\pi$. Then, for all $X^{\prime}\in L_{2}(0,1)$, we have by (3.17) that $f(X^{\prime})={\mathfrak{f}}(\mu_{X^{\prime}})\geq{\mathfrak{f}}(\mu_{X})+{\mathbb{E}}[G(X^{\prime}-X)]=f(X)+{\mathbb{E}}[G(X^{\prime}-X)]$. Thus, $(X,G)\in\partial f$. By Proposition 3.11, $\mu_{(X,G)}\in{\mathfrak{D}}f={\mathfrak{D}}{\mathfrak{f}}$. ∎

Using the characterization of Proposition 3.12, we can, in the spirit of [Roc66], identify which subsets of ${\mathcal{P}}_{2}(\mathbb{R}^{2})$ are contained in the Wasserstein subdifferential of some function ${\mathfrak{f}}\in{\mathfrak{F}}$, or equivalently, of some function $f\in{\mathcal{F}}$. In fact, it is exactly those subsets ${\mathfrak{R}}\subset{\mathcal{P}}_{2}(\mathbb{R}^{2})$ which are jointly cyclically-monotone, as defined in Definition 2.2.

In our first step towards showing that joint cyclic-monotonicity characterizes Wasserstein subdifferentials, we establish that joint cyclic monotonicity of ${\mathfrak{R}}$ imposes the same structure on the distributions $\pi\in{\mathfrak{R}}$ which was identified in Propositions 3.6 to hold for the distributions $\pi_{\mathrm{res}}$.

Lemma 3.13**.**

If ${\mathfrak{R}}$ is jointly cyclically-monotone, then for all $\pi\in{\mathfrak{R}}$ we have

1. (a)

$\pi$* is the optimal coupling between its marginals.* 2. (b)

If $(X,G)\sim\pi$, then $\mu_{(X+G,X)}$ is the optimal coupling between its marginals, and there exists function $\eta\in{\mathcal{PR}}_{1}$ such that $X=\eta\circ(X+G)$.

Proof of Lemma 3.13.

1. (a)

By [Vil10, Theorem 5.10.(ii)], this is equivalent to showing that $\mathsf{spt}(\pi)$ is a cyclically monotone subset of $\mathbb{R}^{2}$. Assume otherwise. Then take $(x,g),(x^{\prime},g^{\prime})\in\mathsf{spt}(\pi)$ such that $x<x^{\prime}$ and $g>g^{\prime}$. Denote by $B_{\epsilon}(x,g)$ the ball of radius $\epsilon$ in $\mathbb{R}^{2}$ centered at $(x,g)$. For some $\epsilon>0$ sufficiently small, we have $\tilde{x}<\tilde{x}^{\prime}$ and $\tilde{g}>\tilde{g}^{\prime}$ whenever $(\tilde{x},\tilde{g})\in B_{\epsilon}(x,g)$ and $(\tilde{x}^{\prime},\tilde{g}^{\prime})\in B_{\epsilon}(x^{\prime},g^{\prime})$. Moreover, $\pi(B_{\epsilon}(x,g)),\,\pi(B_{\epsilon}(x^{\prime},g^{\prime}))>0$ by the definition of the support. Define probability measure $\rho$ to be $\pi|_{B_{\epsilon}(x,g)}/\pi(B_{\epsilon}(x,g))$ and similarly for $\rho^{\prime}$. Take $0<m<\min\{\pi(B_{\epsilon}(x,g)),\,\pi(B_{\epsilon}(x^{\prime},g^{\prime}))\}$ and take $(X,G)\sim\pi$ ,$U\sim\mathsf{Unif}([0,1])$, $(X_{1},G_{1})\sim\rho$, and $(X_{2},G_{2})\sim\rho^{\prime}$, all independent. Define

[TABLE]

It is not hard to verify that $(X,G,X^{\prime},G^{\prime})$ couples $\pi$ to itself. Moreover, $(X-X^{\prime})(G-G^{\prime})$ is equal to zero except in two cases. First, if $(X,G)\in B_{\epsilon}(x,g)\text{ and }U\leq m/\pi(B_{\epsilon}(x,g))$, then it is equal to $(X-X_{2})(G-G_{2})<0$ on this event. Second, if $(X,G)\in B_{\epsilon}(x^{\prime},g^{\prime})\text{ and }U\leq m/\pi(B_{\epsilon}(x^{\prime},g^{\prime}))$, then it is equal to $(X-X_{1})(G-G_{1})<0$ on this event. Thus, ${\mathbb{E}}[(X-X^{\prime})(G-G^{\prime})]<0$, which rearranges to

[TABLE]

a contradiction. Thus, $\mathsf{spt}(\pi)$ is cyclically monotone. 2. (b)

Because $(x,g)\mapsto(x,x+g)$ is a homeomorphism, we have $\mathsf{spt}(\mu_{(X,X+G)})=\{(x,x+g)\mid(x,g)\in\mathsf{spt}(\pi)\}$. In particular, there do not exist $(x,y),(x^{\prime},y^{\prime})\in\mathsf{spt}(\mu_{(X,X+G)})$ for which $x<x^{\prime}$ and $y>y^{\prime}$ because otherwise there exists $(x,g),(x^{\prime},g^{\prime})\in\mathsf{spt}(\pi))$ for which $x<x^{\prime}$ and $g>g^{\prime}$, contradicting the cyclic monotonicity of $\mathsf{spt}(\pi)$. Thus, $\mathsf{spt}(\mu_{(X,X+G)})$ is cyclically monotone. The coupling $\mu_{(X,X+G)}$ thus has the same structure that allowed us to conclude it was implemented by a non-decreasing and 1-Lipschitz mapping applied to $X+G$ in the proof of Proposition 3.7, and the proof proceeds as there.

The proof is complete. ∎

Lemma 3.13 suggests that joint cyclic monotonicity correctly characterizes Wasserstein subdifferentials. The next proposition confirms this, providing a characterization analogous to that in [Roc66] for arbitrary lsc, proper, convex functions. First, we extend Definition 2.2 slightly.

Definition 3.14** (Maximal joint cyclic monotonicity).**

A subset ${\mathfrak{R}}\subset{\mathcal{P}}_{2}(\mathbb{R}^{2})$ is maximally jointly cyclically-monotone if it is jointly cyclically monotone (see Definition 2.2) and not a proper subset of another jointly cyclically monotone set.

Proposition 3.15**.**

Consider any ${\mathfrak{f}}\in{\mathfrak{F}}$ and $f\in{\mathcal{F}}$. Then ${\mathfrak{D}}{\mathfrak{f}}$ and ${\mathfrak{D}}f$ are maximally jointly cyclically monotone. Conversely, if ${\mathfrak{R}}\subset{\mathcal{P}}_{2}(\mathbb{R}^{2})$ is maximally jointly cyclically monotone, then it is the Wasserstein subdifferential of an ${\mathfrak{f}}\in{\mathfrak{F}}$ and an $f\in{\mathcal{F}}$ which are unique up to an additive constant. Further, if ${\mathfrak{R}}$ is jointly cyclically monotone, it is contained in the Wasserstein subdifferential of some ${\mathfrak{f}}\in{\mathfrak{F}}$ and $f\in{\mathcal{F}}$.

The proof of Proposition 3.15 is provided in Appendix B.4.

3.6 Finite dimensional penalties

Having completed our development of symmetric functions on $L_{2}(0,1)$ and their relation to functions defined on Wasserstein space, we are ready to establish that any lsc, proper, symmetric, convex function $f_{p}:\mathbb{R}^{p}\rightarrow\bar{\mathbb{R}}$ can be viewed as the restriction of a function $f\in{\mathcal{F}}$ to a certain set of discrete random variables after embedding $\mathbb{R}^{p}$ into $L_{2}(0,1)$ in a particular way. This embedding idea has also appeared previously in [Day73, Example 4.4].

We will denote the space of lsc, proper, symmetric, convex functions on $\mathbb{R}^{p}$ by ${\mathcal{F}}_{p}$. Let $I_{1},\ldots,I_{p}$ be a partition of $(0,1)$ such that each $I_{j}$ has Lebesgue measure $1/p$, and let ${\mathcal{I}}_{p}$ be the $\sigma$-algebra generated by this partition. Consider the embedding $\iota:\mathbb{R}^{p}\rightarrow L_{2}(0,1)$ defined by

[TABLE]

The embedding $\iota$ is a linear isomorphism between $\mathbb{R}^{p}$ and the ${\mathcal{I}}_{p}$-measurable random variables in $L_{2}(0,1)$. Moreover, it is clear that

[TABLE]

Under this embedding, the function $f\in{\mathcal{F}}$ induces a function $f_{p}\in{\mathcal{F}}_{p}$ defined by777We have selected this normalization to match the relationship between $\|{\boldsymbol{x}}-{\boldsymbol{x}}^{\prime}\|^{2}$ and $d_{2}(\iota({\boldsymbol{x}}),\iota({\boldsymbol{x}}^{\prime}))^{2}$. As a result, later formulas involving proximal operators will not involve annoying factors of $p$.

[TABLE]

Because $\iota$ is linear, continuous, and bijective, $f_{p}$ is indeed lsc, proper, and convex. By (3.19), $f_{p}$ is also symmetric, whence $f_{p}\in{\mathcal{F}}_{p}$ as claimed. We observe that $f_{p}$ does not depend upon the particular embedding $\iota$ of $\mathbb{R}^{p}$ into $L_{2}(0,1)$. Indeed, by (3.19) and (3.2), Eq. (3.22) is equivalent to

[TABLE]

for the unique ${\mathfrak{f}}\in{\mathfrak{F}}$ related to $f$ by (3.2). The right-hand side of (3.21) does not depend on $\iota$.

Definition 3.16** ($L_{2}$ and Wasserstein embeddings).**

The function $f\in{\mathcal{F}}$ is an $L_{2}$ embedding of $f_{p}\in{\mathcal{F}}_{p}$ if (3.20) holds for all isomorphisms $\iota:\mathbb{R}^{p}\rightarrow L_{2}(0,1)$ of the form (3.18). The function ${\mathfrak{f}}\in{\mathfrak{F}}$ is a Wasserstein embedding of $f_{p}\in{\mathcal{F}}_{p}$ if (3.21) holds.

The discussion up to this point establishes the following claim.

Claim 3.17**.**

The function $f\in{\mathcal{F}}$ is an $L_{2}$-embedding of $f_{p}\in{\mathcal{F}}_{p}$ if and only if the function ${\mathfrak{f}}\in{\mathfrak{F}}$ is a Wasserstein embedding of $f_{p}$ for the unique ${\mathfrak{f}}\in{\mathfrak{F}}$ satisfying (3.2).

The next proposition shows that $L_{2}$ and Wasserstein embeddings preserve the structure of proximal operators and their relation to effective scalar representations. Moreover, such embeddings always exist.

Proposition 3.18**.**

Consider $f_{p}\in{\mathcal{F}}_{p}$.

1. (a)

The following are equivalent.

1. (i)

The function $f\in{\mathcal{F}}$ is such that $f+c$ is an $L_{2}$ embedding of $f_{p}$ for some $c\in\mathbb{R}$. 2. (ii)

For all isomporphisms $\iota:\mathbb{R}^{p}\rightarrow L_{2}(0,1)$ of the form (3.18),

[TABLE] 3. (iii)

There exists an isomporphism $\iota:\mathbb{R}^{p}\rightarrow L_{2}(0,1)$ of the form (3.18) for which (3.22) holds. 4. (iv)

For all effective scalar representations ${\mathfrak{A}}$ of $f$,

[TABLE]

where it is understood that ${\mathfrak{A}}(\mu_{{\boldsymbol{y}}})$ is applied coordinate-wise. 5. (v)

There exists an effective scalar representation ${\mathfrak{A}}$ of $f$ such that (3.23) holds. 2. (b)

The following are equivalent.

1. (i)

The function ${\mathfrak{f}}\in{\mathcal{F}}$ is such that ${\mathfrak{f}}+c$ is a Wasserstein embedding of $f_{p}$ for some $c\in\mathbb{R}$. 2. (ii)

For all effective scalar representations ${\mathfrak{A}}$ of ${\mathfrak{f}}$, Eq. (3.23) holds. 3. (iii)

There exists an effective scalar representation ${\mathfrak{A}}$ of ${\mathfrak{f}}$ such that (3.23) holds. 3. (c)

There exists an $L_{2}$ embedding $f$ and a Wasserstein embedding ${\mathfrak{f}}$ of $f_{p}$.

Note that (3.22) makes sense because by Proposition 3.9.(b), $\mathsf{prox}[f_{p}](\iota({\boldsymbol{y}}))$ is guaranteed to be ${\mathcal{I}}_{p}$-measurable, so is in the domain of $\iota^{-1}$.

Proof of Proposition 3.18.

1. (a)

We prove a cycle of implications.

(i) $\Rightarrow$ (ii): Consider any isomorphism $\iota:\mathbb{R}^{p}\rightarrow L_{2}(0,1)$ of the form (3.18). By (3.21), for any ${\boldsymbol{x}}\in\mathbb{R}^{p}$

[TABLE]

By the uniqueness of the minimizer in (1.3), we have (3.23).

(ii) $\Rightarrow$ (iii): This is trivial.

(iii) $\Rightarrow$ (iv): Consider an effective scalar representation ${\mathfrak{A}}$ of $f$. Then for all ${\boldsymbol{y}}\in\mathbb{R}^{p}$ and $i=1,\ldots,p$,

[TABLE]

(iv) $\Rightarrow$ (v): We only need to verify the existence of an effective scalar representation ${\mathfrak{A}}$ of $f$. This holds by Proposition 3.7.

(v) $\Rightarrow$ (i): Let ${\mathfrak{A}}$ be an effective scalar representation of $f$. Define $f_{p}^{\prime}\in{\mathcal{F}}_{p}$ to satisfy (3.20), so that by definition $f$ is an $L_{2}$ embedding of $f_{p}^{\prime}$. Then, as we have already shown, this implies that (3.23) holds with $f_{p}^{\prime}$ in place of $f_{p}$. Thus, for all ${\boldsymbol{y}}\in\mathbb{R}^{p}$, we have $\mathsf{prox}[f_{p}]({\boldsymbol{y}})=\mathsf{prox}[f_{p}^{\prime}]({\boldsymbol{y}})$. From the KKT conditions for the minimization (1.3), we have $({\boldsymbol{x}},{\boldsymbol{g}})\in\partial f_{p}$ if and only if ${\boldsymbol{x}}=\mathsf{prox}[f_{p}]({\boldsymbol{x}}+{\boldsymbol{g}})$, and similarly for $f_{p}^{\prime}$. Thus, the agreement of the proximal operators implies the agreement of the subdifferentials of $f_{p}$ and $f_{p}^{\prime}$. By [Roc66, Theorem 3], this implies the $f_{p}$ and $f_{p}^{\prime}$ agree up to an additive constant. 2. (b)

By Claim 3.17, (i) is equivalent to the function $f$ which satisfies (3.2) being an $L_{2}$ embedding of $f_{p}$. Because by Definition 3.8 the effective scalar representations of ${\mathfrak{f}}$ are exactly the effective scalar representations of $f$, all claimed equivalences follow from part (a). 3. (c)

Consider the subdifferential $\partial f_{p}=\{({\boldsymbol{x}},{\boldsymbol{g}})\in\mathbb{R}^{p}\times\mathbb{R}^{p}\mid{\boldsymbol{g}}\in\partial f_{p}({\boldsymbol{x}})\}$. By symmetry, the membership of $({\boldsymbol{x}},{\boldsymbol{g}})$ in $\partial f_{p}$ depends only upon the joint distribution $\frac{1}{p}\sum_{j=1}^{p}\delta_{(x_{j},g_{j})}$. (Note that all permutations of the coordinates are invertible, so that we do not face the difficulties of Example 3.1). We claim that

[TABLE]

is jointly cyclically monotone. This claim is proved in Appendix B.5.

By Proposition 3.15 and (3.16), there exists $f\in{\mathcal{F}}$ such that ${\mathfrak{R}}\subset{\mathfrak{D}}f$. For any ${\boldsymbol{y}}\in\mathbb{R}^{p}$, note that by the KKT conditions for minimization (1.3), we have ${\boldsymbol{y}}-\mathsf{prox}[f_{p}]({\boldsymbol{y}})\in\partial f_{p}(\mathsf{prox}[f_{p}]({\boldsymbol{y}}))$. If $(X,Y)\sim\mu_{(\mathsf{prox}[f_{p}]({\boldsymbol{x}}),{\boldsymbol{y}})}$, then $\mathsf{prox}[f](Y)=X$ because $\mu_{(X,Y-X)}=\mu_{(\mathsf{prox}[f_{p}]({\boldsymbol{y}}),{\boldsymbol{y}})}\in{\mathfrak{R}}\subset{\mathfrak{D}}f$. Thus, for any isophorphism $\iota:\mathbb{R}^{p}\rightarrow L_{2}(0,1)$ of the form (3.18), we have (3.22). By part (a), $f$ is, up to a constant, an $L_{2}$ embedding of $f_{p}$. Subtracting the constant yields an $L_{2}$ embedding of $f_{p}$. Defining ${\mathfrak{f}}$ via (3.2) yields a Wasserstein embedding by Claim 3.17.

The proof is complete. ∎

4 Proofs of main results

The theory developed in Section 3 allows us to establish the results in Section 2 by constructing $L_{2}$ and Wasserstein embeddings and studying the convergence of empirical measures in Wasserstein space. Here we provide proofs of the results in Section 2 using this theory. Sometimes we will prove more general results than those stated in Section 3 and which may serve as more powerful starting points for future developments. We defer some technical details to the appendix.

4.1 Proof of Theorem 2

Theorem 2 is a particular case of a general result which can be used to study estimators in arbitrary statistical models on $\mathbb{R}^{p}$.

Proposition 4.1**.**

Consider any $f_{p}\in{\mathcal{F}}_{p}$ and $\mu\in{\mathcal{P}}_{2}(\mathbb{R})$. For any Wasserstein embedding ${\mathfrak{f}}$ (resp. $L_{2}$ embedding $f$) of $f_{p}$ and effective scalar representation ${\mathfrak{A}}$ of ${\mathfrak{f}}$ (resp. $f$), we have that

[TABLE]

where it is understood that ${\mathfrak{A}}(\mu)$ is applied to ${\boldsymbol{y}}$ coordinate-wise.

Proof of Proposition 4.1.

By Lemma 3.4, let $Y_{\mathrm{emp}},Y\in L_{2}(0,1)$ be such that $(Y_{\mathrm{emp}},Y)\sim\pi_{\mathrm{opt}}(\mu_{{\boldsymbol{y}}},\mu)$. Let $\iota:\mathbb{R}^{p}\rightarrow L_{2}(0,1)$ be any isomorphism between $\mathbb{R}^{p}$ and the $\sigma(Y_{\mathrm{emp}})$-measurable random variables in $L_{2}(0,1)$ of the form (3.18). Let $f$ be related to ${\mathfrak{f}}$ by (3.2). Then

[TABLE]

where in the first equality we have used (3.22) and that $\iota$ is an isomorphism, in the first inequality we have used the triangle inequality, in the second inequality we have used that $\mathsf{prox}[f]$ is 1-Lipschitz (Proposition 3.9.(a)) and (3.11), and in the last inequality we have used that ${\mathfrak{A}}(\mu)$ is 1-Lipschitz. ∎

We are ready to prove Theorem 2.

Proof of Theorem 2.

By a known concentration inequality on the empirical distribution of Gaussian observations of a fixed parameter ${\boldsymbol{\theta}}$ [MM18, Proposition F.2] (after rescaling by $\tau$), there exist universal functions $\mathsf{c},\mathsf{C}:\mathbb{R}_{>0}\rightarrow\mathbb{R}_{>0}$, non-increasing and non-decreasing respectively, such that for all $\epsilon\in(0,1/2]$,

[TABLE]

where $C=\mathsf{C}(\mathsf{snr})$ and $c=\mathsf{c}(\mathsf{snr})$. Moreover, by Proposition 3.18.(c), for each $p$ and $f_{p}\in{\mathcal{F}}_{p}$ we may choose a Wasserstein embedding ${\mathfrak{f}}$ of $f_{p}$, and by Proposition (3.7), we may then choose an effective scalar representation of ${\mathfrak{f}}$ which we denote by ${\mathfrak{A}}_{f_{p}}$. We prove the theorem for these choices of $\mathsf{c},\mathsf{C}$, and $f_{p}$.

Observe that $W_{2}(\mu_{{\boldsymbol{\theta}}}^{*\tau},\mu^{*\tau})\leq W_{2}(\mu_{{\boldsymbol{\theta}}},\mu)$ because we may consider the coupling between $\mu_{{\boldsymbol{\theta}}}^{*\tau}$ and $\mu^{*\tau}$ constructed as follows: by Lemma 3.4.(b), construct $(X,X^{\prime},G)$ such that $(X,X^{\prime})\sim\pi_{\mathrm{opt}}(\mu_{{\boldsymbol{\theta}}},\mu)$ and $G\sim{\sf N}(0,1)$ independent of $(X,X^{\prime})$. Then $(X+\tau G,X^{\prime}+\tau G)$ couples $\mu_{{\boldsymbol{\theta}}}^{*\tau}$ and $\mu^{*\tau}$ in such a way that ${\mathbb{E}}[(X+\tau G-X^{\prime}-\tau G)^{2}]={\mathbb{E}}[(X-X^{\prime})^{2}]=W_{2}(\mu_{{\boldsymbol{\theta}}},\mu)^{2}$. By Proposition 4.1,

[TABLE]

Combining (4.3) and (4.4) yields (2.2). ∎

Note that Proposition 4.1 can generate results similar to Theorem 2 in alternative statistical models on $\mathbb{R}^{p}$. It quantifies the discrepancy between a symmetric and separable proximal operator in terms of the distance in Wasserstein space between $\mu_{{\boldsymbol{y}}}$ and some fixed $\mu$. Thus, for any statistical model in which $\mu_{{\boldsymbol{y}}}$ concentrates in Wasserstein space, results like Theorem 2, with different rates of convergence, will apply. In particular, if ${\boldsymbol{y}}$ is a possibly random vector in $\mathbb{R}^{p}$, then Proposition 4.1 yields

[TABLE]

To use (4.5), the analyst only needs to bound the right-hand side; that is, she must study concentration rates in Wasserstein space, which depends only on her model and not he choice of penalty $f_{p}$.888Of course, even faster concentration is plausible and may depend upon the choice $f_{p}$. Conveniently, the concentration of empirical measures in Wasserstein space has been extensively studied, see e.g. [FG15].

Moreover, the application of Proposition 4.1 is not limited to concentration results. For example, it can just as easily generate bounds on the expected discrepancy between the symmetric and separable proximal operator if the expected value of $W_{2}(\mu_{{\boldsymbol{y}}},\mu)^{2}$ can be controlled in the statistical model of interest. Even in models in which $\mu_{{\boldsymbol{y}}}$ does not concentrate in Wasserstein space, Proposition 4.1 may generate stochastic information on the behavior of $\mathsf{prox}[f_{p}]$.

4.2 Proof of Theorem 4

Proof of Theorem 4.

First, we will show that

[TABLE]

By (2.5), we must show that for any $\{f_{p}\}\in{\mathcal{C}}_{p}$,

[TABLE]

Consider $\{f_{p}\}\in{\mathcal{C}}_{p}$. Using that $\mathsf{prox}[f_{p}]$ is 1-Lipschitz and the triangle inequality, we have that $\|\mathsf{prox}[f_{p}]({\boldsymbol{\theta}}+\tau{\boldsymbol{z}})-{\boldsymbol{\theta}}\|\geq\|\mathsf{prox}[f_{p}]({\boldsymbol{0}})\|-2\|{\boldsymbol{\theta}}\|-\tau\|{\boldsymbol{z}}\|$. Then

[TABLE]

Because $\frac{1}{\sqrt{p}}{\mathbb{E}}[(2\|{\boldsymbol{\theta}}\|+\tau\|{\boldsymbol{z}}\|)]=O(1)$, if $\frac{1}{p}\|\mathsf{prox}[f_{p}]({\boldsymbol{0}})\|^{2}\rightarrow\infty$, then also $\frac{1}{p}{\mathbb{E}}[\|\mathsf{prox}[f_{p}]({\boldsymbol{\theta}}+\tau{\boldsymbol{z}})-{\boldsymbol{\theta}}\|^{2}]\rightarrow\infty$. Thus, (4.7) is trivial if $\frac{1}{p}\|\mathsf{prox}[f_{p}]({\boldsymbol{0}})\|^{2}\rightarrow\infty$. Thus, without loss of generality, we may assume $\liminf_{p\rightarrow\infty}\frac{1}{p}\|\mathsf{prox}[f_{p}]({\boldsymbol{0}})\|<\infty$. In fact, by the same consideration, we may restrict ourselves to the subsequence $\{p_{\ell}\}$ such that $\frac{1}{p}\|\mathsf{prox}[f_{p_{\ell}}]({\boldsymbol{0}})\|^{2}<M$ for some fixed $M$ sufficiently large that this subsequence is infinite. For simplicity and with some abuse of notation, we denote this subsequence as $\{p\}$.

For each $p$, let ${\mathfrak{f}}^{(p)}$ be a Wasserstein embedding of $f_{p}$ as permitted by Proposition 3.18.(c), and let ${\mathfrak{A}}^{(p)}$ be an effective scalar representation of ${\mathfrak{f}}^{(p)}$ as permitted by Proposition 3.7. Denote ${\boldsymbol{y}}={\boldsymbol{\theta}}+\tau{\boldsymbol{z}}$. An application of the triangle inequality and a few applications of Cauchy-Schwartz yields

[TABLE]

We will show that

[TABLE]

whence it follows that

[TABLE]

To show (4.10), we may assume without loss of generality that the models are all defined on the same probability space and are independent for different values of $p$ because these assumptions do not affect the values of expectations. Because the coordinates of ${\boldsymbol{y}}$ are samplied iid from $\mu^{*\tau}$, by [BF81, Lemma 8.4], we have that

[TABLE]

By Proposition 4.1, we have that

[TABLE]

Eq. (4.10) will follow if we can establish that $\frac{1}{p}\|\mathsf{prox}[f_{p}]({\boldsymbol{y}})-{\mathfrak{A}}^{(p)}(\mu^{*\tau})({\boldsymbol{y}})\|^{2}$ is uniformly integrable. By the triangle inequality and a standard bound on the square of a sum

[TABLE]

Because $\frac{6}{p}\|{\boldsymbol{y}}\|^{2}$ is the empirical mean of iid integrable random variables, it is uniformly integrable over $p$. Also, we established above that along the subsequence we have chosen, $\frac{3}{p}\|\mathsf{prox}[f_{p}]({\boldsymbol{0}})\|^{2}$ is uniformly bounded. Finally, letting $s_{2}(\mu)$ denote the second moment of a distribution $\mu$, we have $\frac{1}{\sqrt{p}}\|{\mathfrak{A}}^{(p)}(\mu^{*\tau})({\boldsymbol{0}})\|\leq s_{2}(\mathsf{prox}[{\mathfrak{f}}^{(p)}](\mu^{*\tau}))+s_{2}(\mu^{*\tau})\leq s_{2}(\mathsf{prox}[{\mathfrak{f}}^{(p)}](\mu_{0}))+2s_{2}(\mu^{*\tau})=\frac{1}{\sqrt{p}}\|\mathsf{prox}[f_{p}]({\boldsymbol{0}})\|+2s_{2}(\mu^{*\tau})$, which we have already shown is uniformly bounded over $p$. We conclude that $\frac{1}{p}\|\mathsf{prox}[f_{p}]({\boldsymbol{y}})-{\mathfrak{A}}^{(p)}(\mu^{*\tau})({\boldsymbol{y}})\|^{2}$ is uniformly integrable, as desired. Thus, we have (4.10), and hence by (4.9) we conclude (4.11). Because ${\mathfrak{A}}^{(p)}(\mu^{*\tau})\in{\mathcal{PR}}_{1}$, the right-hand side of (4.11) is bounded below by $\inf_{f_{1}\in{\mathcal{F}}_{1}}{\mathbb{E}}[(\mathsf{prox}[f_{1}](\theta+\tau z)-\theta)^{2}]=\mathsf{R_{sep}^{opt}}(\tau;\mu)$, as desired. Thus, we have (4.7), hence (4.6).

Now we must show that (4.6) holds with equality. To do so, let $\{f_{1}^{(p)}\}$ be a sequence of functions in ${\mathcal{F}}_{1}$ such that ${\mathbb{E}}[(\mathsf{prox}[f_{1}^{(p)}](\theta+\tau z)-\theta)^{2}]\rightarrow\inf_{f_{1}\in{\mathcal{F}}_{1}}{\mathbb{E}}[(\mathsf{prox}[f_{1}](\theta+\tau z)-\theta)^{2}]$. Then, for $f_{p}({\boldsymbol{x}})=\sum_{j=1}^{p}f_{1}^{(p)}(x_{j})$, we have $\frac{1}{p}{\mathbb{E}}\left[\|\mathsf{prox}[f_{p}]({\boldsymbol{\theta}}+\tau{\boldsymbol{z}})-{\boldsymbol{\theta}}\|^{2}\right]={\mathbb{E}}[(\mathsf{prox}[f_{1}^{(p)}](\theta+\tau z)-\theta)^{2}]$, whence the infimum is attained. This completes the proof. ∎

4.3 Proof of Theorem 5

The proof of Theorem 5 depends upon Gaussian comparison inequalities, using techniques developed in [Sto13, TOH15, TAH16, MM18]. Similar to the approach developed in Section 3, our analysis will rely on embedding a finite-dimensional optimization problem into an optimization problem on $L_{2}(0,1)$.

It is convenient to introduce the variable ${\boldsymbol{v}}={\boldsymbol{b}}-{\boldsymbol{\theta}}$. We may solve (1.4) by instead solving the reparameterized optimization problem

[TABLE]

so that $\widehat{\boldsymbol{\theta}}={\boldsymbol{\theta}}+\widehat{\boldsymbol{v}}$. We denote the objective in (4.12) by $C({\boldsymbol{v}})$. Define

[TABLE]

where ${\boldsymbol{h}}\sim\mathsf{N}({\boldsymbol{0}},{\boldsymbol{I}}_{n})$ and ${\boldsymbol{g}}\sim\mathsf{N}({\boldsymbol{0}},{\boldsymbol{I}}_{p})$, whenever the argument to the square-root is non-negative, and infinity otherwise. We refer to minimizing $L({\boldsymbol{v}})$ as Gordon’s optimization problem because it can be related to the optimization (4.12) via an improvement of Gordon’s theorem [Gor88] found in [TOH15]. Our main tool is the following lemma, which is a slight modification of Corollary 5.1 in [MM18].

Lemma 4.2** (Modification of Corollary 5.1 of [MM18]).**

1. (a)

Let $D\subset\mathbb{R}^{p}$ be a closed set. For all $t\in\mathbb{R}$ we have

[TABLE] 2. (b)

Let $D\subset\mathbb{R}^{p}$ be a convex closed set. We have for all $t\in\mathbb{R}$

[TABLE]

Lemma 4.2 differs from [MM18, Corollary 5.1] only in that we do not assume Gaussian noise ${\boldsymbol{w}}$ as those authors do, which impacts only the form of the objective $L({\boldsymbol{v}})$. In Appendix A, we describe how to derive this objective. We refer the reader to [Gor88, TOH15, TAH18, MM18] for a detailed description of the Gaussian comparison techniques from which Lemma 4.2 follows. Lemma 4.2 allows us to extend any concentration inequality characterizing the minimal value of $L({\boldsymbol{v}})$ over a fixed set to a concentration inequality characterizing the minimal value of $C({\boldsymbol{v}})$ over the same set. With appropriate choices of the set $D$, these inequalities can generate detailed information regarding the minimizers of (1.4).

First, we produce a probabilistic bounds for the minimal value of Gordon’s optimization problem over two carefully chosen sets.

Lemma 4.3**.**

There exist universal functions $\mathsf{c_{1}},\mathsf{c_{2}},\mathsf{c_{3}},\mathsf{c_{4}},\mathsf{C},\mathsf{L}:\mathbb{R}_{>0}^{4}\rightarrow\mathbb{R}_{>0}$ such that the following is true.

Consider any $f_{p}\in{\mathcal{F}}_{p}$, ${\boldsymbol{\theta}}\in\mathbb{R}^{p}$ with $\|{\boldsymbol{\theta}}\|^{2}/p\leq M$ for some $M$, and $\sigma^{*},\tau^{*},\lambda^{*},\delta>0$. Assume $f$ is any $L_{2}$ embedding of $pf_{p}$ and ${\mathfrak{A}}$ is any effective scalar representation of $\lambda^{*}f$, and assume that

[TABLE]

where $Y^{*}=\Theta^{*}+\tau^{*}G^{*}$ with $\Theta^{*},G^{*}\in L_{2}(0,1)$, $\Theta^{*}\sim\mu_{{\boldsymbol{\theta}}}$ and $G^{*}\sim\mathsf{N}(0,1)$ independent of $\Theta^{*}$. Define

[TABLE]

Define the random variable $\sigma^{2}=\|{\boldsymbol{w}}\|^{2}/n$ and constants $c_{i}=\mathsf{c_{i}}(M,\tau^{*},\sigma^{*},\delta)$, $C=\mathsf{C}(M,\tau^{*},\sigma^{*},\delta)$, and $L=\mathsf{L}(M,\tau^{*},\sigma^{*},\delta)$. Then in model (1.2) with $n/p=\delta$ and ${\boldsymbol{w}}$ independent of ${\boldsymbol{X}}$, we have for $0<\epsilon<c_{1}$ that

[TABLE]

*where *

[TABLE]

Lemma 4.3 is proved in Appendix B.6. Using Lemma 4.3 we may prove Theorem 5.

Proof of Theorem 5.

Choose $\mathsf{c_{1}},\mathsf{c_{2}},\mathsf{c_{3}},\mathsf{C}$ as in Lemma 4.3 and for each $f_{p}^{\prime}$ choose ${\mathfrak{A}}_{f_{p}^{\prime}}$ to be any effective scalar representation of any $L_{2}$ embedding of $pf_{p}^{\prime}$, which exists by Propositions 3.7 and 3.18.(c).

Consider also $\mathsf{c_{4}},\mathsf{L}$ as in Lemma 4.3. Define $c_{i}=\mathsf{c_{i}}(M,\tau^{*},\sigma^{*},\delta)$, $C=\mathsf{C}(M,\tau^{*},\sigma^{*},\delta)$, and $L=\mathsf{L}(M,\tau^{*},\sigma^{*},\delta)$. By Lemmas 4.2 and 4.3, we have for $0<\epsilon<c_{1}$ that

[TABLE]

Note that when the event in (4.20) fails to occur, by convexity the minimizer of $C({\boldsymbol{v}})$ over $\mathbb{R}^{p}$ falls in $D_{\epsilon}$ (defined in (4.19)). On this event, we have $W_{2}(\mu_{(\widehat{\boldsymbol{\theta}},{\boldsymbol{\theta}})},\mu_{({\mathfrak{A}}_{\lambda^{*}f_{p}}(\mu^{*\tau^{*}})\circ Y^{*},\Theta^{*})})^{2}\leq{\tau^{*}}^{2}\epsilon$. We conclude (2.15). ∎

4.4 Proof of Theorem 6

Fix $p$. Because $\{\mu_{({\mathfrak{A}}(\mu_{Y})\circ Y,Y-{\mathfrak{A}}(\mu_{Y})\circ Y)}\mid\mu_{Y}\in{\mathcal{D}}\}$ is jointly cyclically monotone, we may find $f\in{\mathcal{F}}$ such that $\{\mu_{({\mathfrak{A}}(\mu_{Y})\circ Y,Y-{\mathfrak{A}}(\mu_{Y})\circ Y)}\mid\mu_{Y}\in{\mathcal{D}}\}\subset{\mathfrak{D}}{\mathfrak{f}}$. By Proposition 3.11, this implies that for all $Y\in L_{2}(0,1)$ with $\mu_{Y}\in{\mathcal{D}}$, we have $Y-{\mathfrak{A}}(\mu_{Y})\in\partial f({\mathfrak{A}}(\mu_{Y})\circ Y)$. By the KKT conditions, this implies that

[TABLE]

Now consider ${\mathfrak{A}}^{\prime}$ an effective scalar representation of $f$, which is exist by Proposition 3.7. With ${\mathfrak{A}}^{\prime\prime}={\mathfrak{A}}|_{{\mathcal{D}}}+{\mathfrak{A}}^{\prime}|_{{\mathcal{D}}^{c}}$, we have that (3.11) holds for ${\mathfrak{A}}^{\prime\prime}$ for all $Y\in L_{2}(0,1)$ by applying (4.21) to ${\mathfrak{A}}$ on ${\mathcal{D}}$ and (3.11) to ${\mathfrak{A}}^{\prime}$ on ${\mathcal{D}}^{c}$. By Proposition 3.9.(b), ${\mathfrak{A}}^{\prime\prime}$ is an effective scalar representation of $f$. Now define $f_{p}$ by (3.20), so that $f$ is an $L_{2}$ embedding of $f_{p}$. By considering the same universal functions $\mathsf{c},\mathsf{C}$ of Theorem 2, the result follows by the proof of that theorem and using that ${\mathfrak{A}}^{\prime\prime}|_{{\mathcal{D}}}={\mathfrak{A}}|_{{\mathcal{D}}}$.

Acknowledgements

The author is grateful to Andrea Montanari for encouragement and insightful conversations and comments. This material is based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE – 1656518.

Appendix A Proof of Lemma 4.2

We write the (random) value of the optimization in (4.12) as

[TABLE]

Morover, we claim

[TABLE]

Indeed, if we maximize first over the direction of ${\boldsymbol{u}}$ to get

[TABLE]

If we then maximize over $\beta$, we get $\min_{{\boldsymbol{v}}}L({\boldsymbol{v}})$.

Gordon’s Theorem [Gor88], in which a minmax problem involving a Gaussian matrix ${\boldsymbol{X}}$ is related to a particular minmax problem involving Gaussian vectors ${\boldsymbol{g}},{\boldsymbol{h}}$, permits the comparison of the min-max problems (A.1) and (A.2). Specifically, if we condition on ${\boldsymbol{w}}$ (so that it may be viewed as deterministic), we may follow exactly the proof of [MM18, Corollary 5.1] –with the only difference being that because ${\boldsymbol{w}}$ is not Gaussian we have a different form for Gordon’s optimization problem– to achieve the probability bounds of Lemma 4.2 conditional on ${\boldsymbol{w}}$. Taking expectations over ${\boldsymbol{w}}$, we also have the probability bounds of Lemma 4.2 unconditionally.

We refer the reader to [Gor88, TOH15, TAH18, MM18] for a detailed description of the Gaussian comparison techniques involved in the above argument. The most recent of these, [MM18], includes a mostly self-contained presentation of the relevant results.

Appendix B Proofs of additional technical results

B.1 Proof of Fact 2.1

For $\eta:\mathbb{R}\rightarrow\mathbb{R}$, there exists lsc, proper, convex $\rho$ such that $\eta=\mathsf{prox}[\rho]$ if and only if $\{(\eta(y),y-\eta(y))\mid y\in\mathbb{R}\}$ is cyclically monotone [Roc66, Theorem 1]. This is equivalent to the following: if $\eta(y)<\eta(y^{\prime})$ then $y-\eta(y)\leq y^{\prime}-\eta(y^{\prime})$, and if $y-\eta(y)<y^{\prime}-\eta(y^{\prime})$, then $\eta(y)\leq\eta(y^{\prime})$. This, in turn, is equivalent to $\eta$’s being non-decreasing and 1-Lipschitz, as we now show.

First, consider that $\eta$ is non-decreasing and 1-Lipschitz. Then, if $\eta(y)<\eta(y^{\prime})$, it must be the case that $y<y^{\prime}$, whence $y-\eta(y)=y^{\prime}-\eta(y^{\prime})-((y-y^{\prime})-(\eta(y)+\eta(y^{\prime}))\leq y^{\prime}-\eta(y^{\prime})$. Note that $\eta$ being non-decreasing and 1-Lipschitz is equivalent to $\eta$ and $\mathrm{Id}-\eta$ both being non-decreasing, whence, by the same argument, if $y-\eta(y)<y^{\prime}-\eta(y^{\prime})$, then $\eta(y)\leq\eta(y^{\prime})$.

Next, we prove the converse. First, if $y\leq y^{\prime}$, then $\eta(y)\leq\eta(y^{\prime})$ because otherwise we would have $\eta(y)>\eta(y^{\prime})$ and $y-\eta(y)<y^{\prime}-\eta(y^{\prime})$, a contradiction. Thus, $\eta$ is non-decreasing. By symmetry, $\mathrm{Id}-\eta$ is also non-decreasing, whence $\eta$ is 1-Lipschitz as well.

B.2 Proof of Lemma 3.4

Proof of Lemma 3.4.

By [Kal02, Lemma 3.21], there exists $U,U_{1},U_{2},\ldots$ independent, uniformly distributed random variables in $L_{2}(0,1)$.

First, we prove part (a). Let $A_{j}$ be the coordinate functions on the probability space $(\mathbb{R}^{k},\pi,{\mathcal{B}})$. By [Kal02, Theorem 3.19], there exists $X_{1}\in L_{2}(0,1)$ with $X_{1}\stackrel{{\scriptstyle\mathrm{d}}}{{=}}A_{1}$, whence also $X_{1}\circ U_{1}\stackrel{{\scriptstyle\mathrm{d}}}{{=}}A_{1}$. By [Kal02, Theorem 6.3], the distribution of $A_{j}$ given $A_{1},\ldots,A_{j-1}$ admits a regular conditional probability distribution for each $j=2,\ldots,k$. Thus, by [Kal02, Lemma 3.22], there exists for each such $j$ a measurable function $\zeta_{j}:\mathbb{R}^{j-1}\times[0,1]\rightarrow\mathbb{R}$ such that

[TABLE]

In particular, if we define $X_{j}$ inductively by $X_{j}=\zeta_{j}(X_{1},\ldots,X_{j-1},\zeta_{j}(X_{1},\ldots,X_{j-1},U_{j}))$ for $j\geq 2$, we have the $(X_{1},\ldots,X_{k})\stackrel{{\scriptstyle\mathrm{d}}}{{=}}(A_{1},\ldots,A_{k})\sim\pi$. Hence, part (a).

Next, we prove part (b). As above, by [Kal02, Theorem 3.19], there exists $X\in L_{2}(0,1)$ with $X\sim\mu$, whence also $X\circ U\sim\mu$. By [Kal02, Theorem 6.3], each coupling $\pi_{p}$ admits regular conditional probability distributions, whence by [Kal02, Lemma 3.22], there exists for each $p$ a measurable function $\zeta_{p}:\mathbb{R}\times[0,1]\rightarrow\mathbb{R}$ such that $(X\circ U,\zeta_{p}(X\circ U,U_{p}))\sim\pi_{p}$. Defining $X_{p}=\zeta_{p}(X\circ U,U_{p})$ yields part (b).

Part (c) follows by applying part (b) to the couplings $\pi_{p}=\pi_{\mathrm{opt}}(\mu,\mu_{p})$ for each $p$ and observing that if $(X,X_{p})\sim\pi_{\mathrm{opt}}(\mu,\mu_{p})$ then $d_{2}(X,X_{p})=W_{2}(\mu,\mu_{p})$. ∎

B.3 Proximal operators on Wasserstein space are well-defined

In this Appendix, we show that when ${\mathfrak{f}}$ is lsc, proper, and convex, the minimizer on the right-hand side of (3.5) exists and is unique.

First, we show that

[TABLE]

is finite. Becuase ${\mathfrak{f}}$ is proper, there exists $\nu$ for which the objective is less than $\infty$, whence we must only show the infimum is not $-\infty$. Take any $\nu$ such that ${\mathfrak{f}}(\nu)<\infty$. By lower semi-continuity, there exists $\epsilon>0$ such that ${\mathfrak{f}}(\nu^{\prime})>{\mathfrak{f}}(\nu)-1$ whenever $W_{2}(\nu^{\prime},\nu)\leq\epsilon$. Fix such an $\epsilon$, and consider $\nu^{\prime}$ with $W_{2}(\nu^{\prime},\nu)>\epsilon$. Observe that

[TABLE]

Thus,

[TABLE]

where the first inequality holds by the definition of $\epsilon$ and the second by convexity of ${\mathfrak{f}}$ (see Definition 3.3). Rearranging, we have

[TABLE]

Thus,

[TABLE]

where the first inequality holds by dividing into cases $W_{2}(\nu^{\prime},\nu)\leq\epsilon$ and $>\epsilon$, the second inequality holds by the triangle inequality, and the remaining (in)equalities hold by algebrabraic manipulation. Thus, (B.1) is finite.

Now consider any $\epsilon\geq 0$ and $\nu^{\prime},\nu^{\prime\prime}$ which achieve objective values within $\epsilon$ of the infimum (here we use finiteness of (B.1)) . By Lemma 3.4.(b), there exists $X,X^{\prime},X^{\prime\prime}\in L_{2}(0,1)$ with $(X,X^{\prime})\sim\pi_{\mathrm{opt}}(\mu,\nu^{\prime})$ and $(X,X^{\prime\prime})\sim\pi_{\mathrm{opt}}(\mu,\nu^{\prime\prime})$. Then

[TABLE]

The parallelogram identity gives

[TABLE]

where the last inequality follows from (B.6). Thus, $W_{2}(\nu^{\prime},\nu^{\prime\prime})^{2}\leq 8\epsilon$.

The argument in the preceding paragraph implies that if $\{\nu_{j}\}$ is a sequence along which the objective value in (B.1) approaches its infimum, the sequence is Cauchy. Because Wasserstein space is complete [Vil10, Theorem 6.18], this sequence has limit $\nu^{*}$. Because the objective is lower-semicontinuous, the infimum is achieved at $\nu^{*}$.

That minimizer of the objective in (B.1) is unique follows by taking $\epsilon=0$ in (B.7).

B.4 Proof of Proposition 3.15

Proof of Proposition 3.15.

Because (3.16) whenever $f$ and ${\mathfrak{f}}$ are related by (3.2), it is enough to prove each part of the proposition for either $f$ or ${\mathfrak{f}}$. At each point of the proof, we work with whichever is most convenient.

Step 1: ${\mathfrak{D}}f$ (and hence ${\mathfrak{D}}f$) are maximally jointly cyclically monotone. First we show that if $f\in{\mathcal{F}}$, then ${\mathfrak{D}}f$ is maximally jointly cyclically monotone. Indeed, for all finite $n$, all sets $\{\pi_{j}\}_{j=1}^{n}\subset{\mathfrak{D}}f$, and all random vectors $(X_{1},G_{1},\ldots,X_{n},G_{n})$ with $(X_{j},G_{j})\sim\pi_{j}$ for all $j$, we have by Proposition 3.11 that $G_{j}\in\partial f(X_{j})$. Then, for all permutations $\sigma:[n]\rightarrow[n]$, we have by the definition of subdifferential that for all $j\in[n]$

[TABLE]

where by convention we denote $\sigma(n+1)=\sigma(1)$. Summing over $j\in[n]$ and rearranging gives (2.16), whence ${\mathfrak{D}}f$ is jointly cyclically monotone. To see that it is maximal, assume for contradiction there were ${\mathfrak{R}}\subset{\mathcal{P}}_{2}(\mathbb{R}^{2})$ strictly containing ${\mathfrak{D}}f$ which was jointly cyclically monotone. Then by Lemma 3.4.(a), we have $\{(X,G)\mid X,G\in L_{2}(0,1),\,\mu_{(X,G)}\in{\mathfrak{R}}\}$ is a strict super-set of $\partial f$. Moreover, by (2.16), it is cyclically monotone. This contradicts Theorem 3 of [Roc66], which states that the subdifferential relation $\partial f$ is always maximally cyclically monotone. Thus, ${\mathfrak{D}}f$ is maximally jointly cyclically monotone.

Step 2: If ${\mathfrak{R}}$ is jointly cyclically monotone, it is contained in the Wasserstein sub-differential of an ${\mathfrak{f}}\in{\mathfrak{F}}$ (and hence $f\in{\mathcal{F}}$). We imitate the proof of [Roc66, Theorem 1], but apply the proof technique to ${\mathfrak{f}}$ instead of $f$.

Pick some $\pi_{0}\in{\mathfrak{R}}$. For all $\mu\in{\mathcal{P}}_{2}(\mathbb{R})$, define

[TABLE]

where the supremum is taken over all positive integers $n$ and all vectors $(X,X_{0},G_{0},\ldots,X_{n},G_{n})$ satsifying $\mu_{X}=\mu$ and $\mu_{(X_{j},G_{j})}=\pi_{j}$ for all $0\leq j\leq n$. We confirm that ${\mathfrak{f}}$ so defined is lsc, proper, and convex, and then will show that ${\mathfrak{R}}\subset{\mathfrak{D}}{\mathfrak{f}}$.

First, we show ${\mathfrak{f}}$ is proper. Let $\mu_{0}$ be the first marginal of $\pi_{0}$. We show that ${\mathfrak{f}}(\mu_{0})<\infty$. Let $\{\pi_{j}\}_{j=1}^{n}\subset{\mathfrak{D}}f$ and $(X,X_{0},G_{0},\ldots,X_{n},G_{n})$ be such that $\mu_{X}=\mu_{0}$ and $\mu_{(X_{j},G_{j})}=\pi_{j}$. Let $F_{X_{j}}$ and $F_{G_{j}}$ denote the cdfs of $X_{j},G_{j}$, respectively. Consider $\tilde{X}_{j},\tilde{G}_{j}\in L_{2}(0,1)$ defined by $\tilde{X}_{j}(t)=F_{X_{j}}^{-1}(t)$ and $\tilde{G}_{j}=F_{G_{j}}^{-1}(t)$ for $t\in(0,1)$. By [Vil10, Theorem 5.10.(ii)], for any $i,j$ we have that $\mu_{(\tilde{X}_{i},\tilde{G}_{j})}$ is the optimal coupling between its marginals because $F_{X_{j}}$ and $F_{G_{j}}$ have the same ordering with respect to $t$, so that the support of $\mathsf{spt}(\mu_{(\tilde{X}_{i},\tilde{G}_{j})})$ so constructed is cyclically monotone (see Definition 3.5). Thus,

[TABLE]

Moreover, by Lemma 3.13.(a), we have that $\mu_{(X_{j},G_{j})}$ is also the optimal coupling between its marginals, whence $\mu_{(\tilde{X}_{j},\tilde{G}_{j})}=\mu_{(X_{j},G_{j})}$ for all $j$. Thus,

[TABLE]

Combining the previous two displays,

[TABLE]

Moreover, the left-hand side is bounded above by [math] by Definition 2.2 because $(\tilde{X}_{j},\tilde{G}_{j})\sim\pi_{j}$ for all $j$ and ${\mathfrak{R}}$ is jointly cyclically monotone. Thus, we conclude that ${\mathfrak{f}}(\mu_{0})\leq 0$.

Second, we show ${\mathfrak{f}}$ is lower semi-continuous. Consider $\mu\in{\mathcal{P}}_{2}(\mathbb{R})$ and ${\mathfrak{f}}(\mu)<\infty$. Let $(X,X_{0},G_{0},\ldots,X_{n},G_{n})$ be as above with

[TABLE]

Now consider any $\mu^{\prime}$ such that $W_{2}(\mu,\mu^{\prime})<\epsilon/\sqrt{{\mathbb{E}}[G_{n}^{2}]}$. Then, by the gluing lemma [Vil10, pg. 11] and Lemma 3.4.(a), we may construct $(X^{\prime},\tilde{X},\tilde{X}_{0},\tilde{G}_{0},\ldots,\tilde{X}_{n},\tilde{G}_{n})$ such that $\mu_{(X^{\prime},\tilde{X})}$ is the optimal coupling between $\mu^{\prime}$ and $\mu$, and $(\tilde{X},\tilde{X}_{0},\tilde{G}_{0},\ldots,\tilde{X}_{n},\tilde{G}_{n})\stackrel{{\scriptstyle\mathrm{d}}}{{=}}(X,X_{0},G_{0},\ldots,X_{n},G_{n})$. By (B.8),

[TABLE]

Because $\epsilon$ was arbitrary, we get lower semi-continuity at any $\mu$ such that ${\mathfrak{f}}(\mu)<\infty$. For ${\mathfrak{f}}(\mu)=\infty$, we instead take the right-hand side of (B.9) to be $N$ and take $N\rightarrow\infty$.

Third, we show ${\mathfrak{f}}$ is convex. Consider any $\mu,\mu^{\prime}\in{\mathcal{P}}_{2}(\mathbb{R})$, any coupling $\pi$ between them, and any $\alpha\in[0,1]$. If ${\mathfrak{f}}(\alpha\mu\oplus_{\pi}(1-\alpha)\mu^{\prime})<\infty$, consider any $\epsilon>0$ and take $(X^{\prime\prime},X_{0},G_{0},\ldots,X_{n},G_{n})$ such that $X^{\prime\prime}\sim\alpha\mu\oplus_{\pi}(1-\alpha)\mu^{\prime}$, $(X_{j},G_{j})\sim\pi_{j}$ for $1\leq j\leq n$ where $\pi_{j}\in{\mathfrak{R}}$ and $j\geq 1$, $(X_{0},G_{0})\sim\pi_{0}$, and

[TABLE]

By the gluing lemma [Vil10, pg. 11] and Lemma 3.4.(a), we can construct $(\tilde{X},\tilde{X}^{\prime},\tilde{X}^{\prime\prime},\tilde{X}_{0},\tilde{G}_{0},\ldots,\tilde{X}_{n},\tilde{G}_{n})$ such that $\tilde{X}^{\prime\prime}=\alpha\tilde{X}+(1-\alpha)\tilde{X}^{\prime}$ (almost surely) and $(\tilde{X}^{\prime\prime},\tilde{X}_{0},\tilde{G}_{0},\ldots,\tilde{X}_{n},\tilde{G}_{n})$ has the same distribution as $(X^{\prime\prime},X_{0},G_{0},\ldots,X_{n},G_{n})$. Then

[TABLE]

Taking $\epsilon\rightarrow 0$ yields the result. If ${\mathfrak{f}}(\alpha\mu\oplus_{\pi}(1-\alpha)\mu^{\prime})=\infty$, we instead take ${\mathbb{E}}[G_{n}(X^{\prime\prime}-X_{n})]+\cdots+{\mathbb{E}}[G_{0}(X_{1}-X_{0})]\geq N$ in place of (B.10) and take $N\rightarrow\infty$ to conclude that either ${\mathfrak{f}}(\mu)$ or ${\mathfrak{f}}(\mu^{\prime})$ is infinite.

Finally, we check that ${\mathfrak{R}}\subset{\mathfrak{D}}{\mathfrak{f}}$. We do this by checking the condition in Proposition 3.12. Consider $\pi\in{\mathfrak{R}}$, and let $\mu$ be its first marginal. Let $\mu^{\prime}\in{\mathcal{P}}_{2}(\mathbb{R})$. Consider $(X,G,X^{\prime})$ with $(X,G)\sim\pi$ and $X^{\prime}\sim\mu$. Fix $\alpha<{\mathfrak{f}}(\mu)$. By (B.8), the gluing lemma [Vil10, pg. 11] and Lemma 3.4.(a), we may assume that we in fact have $(X,G,X^{\prime},X_{0},G_{0},\ldots,X_{n},G_{n})$ such that

[TABLE]

By (B.8),

[TABLE]

Taking $\alpha\uparrow{\mathfrak{f}}(\mu_{X})$ gives (3.17).

Step 3: If ${\mathfrak{R}}$ is maximally jointly cyclically monotone, then it is the Wasserstein subdifferential of an ${\mathfrak{f}}\in{\mathfrak{F}}$ (and hence $f\in{\mathcal{F}}$). By step 2, ${\mathfrak{R}}\subset{\mathfrak{D}}{\mathfrak{f}}$ for some ${\mathfrak{f}}\in{\mathfrak{F}}$. By step 1, ${\mathfrak{D}}{\mathfrak{f}}$ is jointly cyclically monotone. The maximality of ${\mathfrak{R}}$ implies ${\mathfrak{R}}={\mathfrak{D}}{\mathfrak{f}}$.

Step 4: Uniqueness up to an additive constant. The Wasserstein subdifferential of $f$ uniquely determines the subdifferential of $f$ by Proposition 3.11, and the subdifferential of $f$ uniquely determines $f$ up to an additive constant by [Roc66, Theorem 3]. ∎

B.5 Omitted part of proof of Proposition 3.18.(c)

Here we show that ${\mathfrak{R}}$ defined in (3.25) is jointly cyclically monotone, as claimed.

First, we show that $\mathsf{spt}\left(\frac{1}{p}\sum_{i=1}^{p}\delta_{(x_{i},g_{i})}\right)$ is cyclically monotone for any $({\boldsymbol{x}},{\boldsymbol{g}})\in\partial f$. By [Roc66, Theorem 1], we have that for all $({\boldsymbol{x}}_{1},{\boldsymbol{g}}_{1}),\ldots,({\boldsymbol{x}}_{n},{\boldsymbol{g}}_{n})\in\partial f_{p}$ and all permutation $\sigma:[n]\rightarrow[n]$ that

[TABLE]

For any permutation $\tau:[p]\rightarrow[p]$, denote ${\boldsymbol{x}}^{\tau}\in\mathbb{R}^{p}$ the vector with $({\boldsymbol{x}}^{\tau})_{i}={\boldsymbol{x}}_{\tau(i)}$. Then, if we let $\tau$ be the transposition $(i,k)$ for $i\neq k$ and consider any $({\boldsymbol{x}},{\boldsymbol{g}})\in\partial f_{p}$, we also have $({\boldsymbol{x}}^{\tau},{\boldsymbol{g}}^{\tau})\in\partial f_{p}$ by symmetry. Thus,

[TABLE]

which is exactly the inequality $(x_{i}-x_{k})(g_{i}-g_{k})\geq 0$. This is equivalent to the statement that for no $i,k$ is it the case that $x_{i}>x_{k}$ and $g_{i}<g_{k}$. By Definition 3.5, $\mathsf{spt}\left(\frac{1}{p}\sum_{i=1}^{p}\delta_{(x_{i},g_{i})}\right)$ is cyclically-monotone, as desired.

Now consider any $\pi_{1},\ldots,\pi_{n}\in{\mathfrak{R}}$ and $(X_{1},G_{1},\ldots,X_{n},G_{n})$ with $(X_{j},G_{j})\sim\pi_{j}$ for all $j$. We show (3.14) for any permutation $\sigma:[n]\rightarrow[n]$. Consider any permutation $\sigma:[n]\rightarrow[n]$. We may write $\mu_{X_{j}}=\frac{1}{p}\sum_{i=1}^{p}\delta_{x_{j(i)}}$ where $x_{j(1)}\leq x_{j(2)}\leq\cdots\leq x_{j(p)}$ is the ordered enumeration of the support points of $\mu_{X_{j}}$, and similarly for $G_{j}$. Note that $\{(x_{j(i)},g_{\sigma(j)(i)})\mid i=1,\ldots,p\}$ is cyclically monotone because $x_{j(i)}$ and $g_{\sigma(j)(i)}$ share the same ordering with respect to $i$, whence by [Vil10, Theorem 5.10.(ii)], we have $\frac{1}{p}\sum_{i=1}^{p}\delta_{(x_{j(i)},g_{\sigma(j)(i)})}$ is the optimal coupling between $\mu_{X_{j}}$ and $\mu_{G_{\sigma(j)}}$. Thus,

[TABLE]

Also,

[TABLE]

because if we define ${\boldsymbol{x}}_{j},{\boldsymbol{g}}_{j}$ to have coordinates given by $x_{j(i)}$ and $g_{j(i)}$, respectively, then $({\boldsymbol{x}}_{j},{\boldsymbol{g}}_{j})\in\partial f$, whence (B.13) follows from (B.11). Finally, we identify the right-hand side of (B.13) as $\sum_{j=1}^{n}{\mathbb{E}}[X_{j}G_{j}]$ because the support of $\mathsf{spt}(\mu_{(X_{j},G_{j})})$ is cyclically monotone. Compining the previous three displays, we conclude (3.14).

Thus, ${\mathfrak{R}}$ is jointly-cyclically monotone.

B.6 Proof of Lemma 4.3

Our main idea is to study instead an embedding of Gordon’s optimization problem into $L_{2}(0,1)$. For $B,\Theta,G\in L_{2}(0,1)$, $\sigma\in\mathbb{R}_{\geq 0}$, and $\kappa,\xi\in\mathbb{R}$, define

[TABLE]

whenever the arguments to the square-roots are non-negative, and infinite otherwise. We refer the optimization problem

[TABLE]

as the Gordon’s $L_{2}$ optimization. We write the objective in (B.15) as ${\mathcal{L}}(B;\Theta,G,\sigma^{2},\kappa,\xi)$. The proof proceeds in several steps.

Step 1: Convexity of Gordon’s $L_{2}$ objective when $\xi=0$. When $\xi=0$, the arguments to the square-roots in (B.14) are non-negative, whence the objective is defined everywhere. Moreover, as we now show, the objective is also convex in $B$, and locally strongly convex.

Lemma B.1**.**

Consider $\Theta\in L_{2}(0,1)$. Let $h(B)=\sqrt{\sigma^{2}+\frac{1}{\delta}{\mathbb{E}}[(B-\Theta)^{2}]}$. Then $h$ is $\frac{\sigma^{2}/\delta}{(\sigma^{2}+R^{2}/\delta)^{3/2}}$-strongly convex on $d_{2}(B,\Theta)\leq R$.

Proof of Lemma B.1.

Let $B_{t}=B+t\Delta$ and $V_{t}=B_{t}-\Theta$. Then we have

[TABLE]

Then

[TABLE]

where in the inequality, we have used Cauchy-Schwartz. We conclude that $h(B)$ is $\frac{\sigma^{2}/\delta}{(\sigma^{2}+R^{2}/\delta)^{3/2}}$-strongly convex on $d_{2}(B,\Theta)\leq R$. ∎

Step 2: Solution to Gordon’s $L_{2}$ population optimization. We study first the simple case that $\sigma={\sigma^{*}},\kappa=1,\xi=0$, and $\Theta=\Theta^{*},G=G^{*}$ for any $\Theta^{*}\sim\mu_{{\boldsymbol{\theta}}},G^{*}\sim{\sf N}(0,1)$ independent, which we call Gordon’s $L_{2}$ population optimization. For simplicity of notation, we denote $\eta={\mathfrak{A}}(\mu^{*{\tau^{*}}})$. Fixing such $\Theta^{*},G^{*}$, we denote $B^{*}=\eta\circ Y^{*}$. By (4.16),

[TABLE]

By (4.17),

[TABLE]

Because $g$ is positive at $B^{*},\Theta^{*},G^{*},{\sigma^{*}}$, the Frèchet derivative of $g_{+}^{2}$ with respect to its first argument, denoted by $\nabla_{B}$, evaluated at $B^{*},\Theta^{*},G^{*},\sigma^{*}$ is

[TABLE]

where we have used (B.6). By (3.11), $B^{*}=\mathsf{prox}[\lambda^{*}f](\Theta^{*}+\tau^{*}G^{*})$, whence

[TABLE]

Combining the previous two displays, 0\in\partial\left(\frac{1}{2}g(B;\Theta^{*},G^{*},{\sigma^{*}},1,0)_{+}^{2}+f(B)\right)\Big{|}_{B=B^{*}}, and by convexity

[TABLE]

Step 3: Solution to Gordon’s $L_{2}$ partially perturbed optimization. If we only perturb $\Theta$, $G$, and $\sigma$, so that still $\kappa=1$ and $\xi=0$, the objective remains convex. Moreover, in this case the solution to (B.15) satisfies the following.

Lemma B.2**.**

If $\kappa=1,\xi=0$ and a solution $B_{\mathrm{opt}}$ to (B.15) is such that $g(B_{\mathrm{opt}};\Theta,G,\sigma,1,0)>0$, then $B_{\mathrm{opt}}=\mathsf{prox}\left[\lambda f\right](\Theta+\tau G)$, where $\tau^{2}=\sigma^{2}+\frac{1}{\delta}{\mathbb{E}}[(B_{\mathrm{opt}}-\Theta)^{2}]$ and $\lambda^{-1}=\frac{1}{\delta}\left(1-\frac{1}{\delta\tau}{\mathbb{E}}[G(B_{\mathrm{opt}}-\Theta)]\right)$. In particular, $B_{\mathrm{opt}}$ is $\sigma(\Theta+\tau G)$-measurable.

Proof.

We have

[TABLE]

where in the second equality we have used the definition of $\tau$, and in the third equality we have used the definition of $\lambda$. The KKT conditions give

[TABLE]

whence $B_{\mathrm{opt}}=\mathsf{prox}[\lambda f](\Theta+\tau G)$. Measurability follows from Proposition 3.9. ∎

Step 4: Derivatives of $\tau$. We will control solutions to Gordon’s $L_{2}$ optimization problem by viewing Gordon’s $L_{2}$ optimization as a perturbation of Gordon’s $L_{2}$ population optimization and controlling the sensitivity of the solutions to perturbations in the problem. This requires the study of several derivates. Define

[TABLE]

whenever the argument to the square-root is positive. For any variable $A$, we denote the Frèchet derivative with respect to $A$ by $\nabla_{A}$. We have

[TABLE]

Step 5: Lipschitz continuity and strong convexity parameters. We define several local Lipschitz and strong converxity parameters which are entirely explicit in $\tau^{*},\lambda^{*},\delta$. Let

[TABLE]

By (4.16),

[TABLE]

The following will serve as local Lipschitz constants:

[TABLE]

and

[TABLE]

and

[TABLE]

The following will enter our definition of “local”:

[TABLE]

and for $A\in\{\Theta,B,G,\sigma,\kappa,\xi\}$ (viewed as a symbol and a collection of symbols), let

[TABLE]

For ${\boldsymbol{r}}=(r^{(\Theta)},r^{(B)},r^{(G)},r^{(\sigma)},r^{(\kappa)},r^{(\xi)})$, we say $(\Theta,B,G,\sigma,\kappa,\xi)\in N_{{\boldsymbol{r}}}$ if

[TABLE]

With some abuse of notation, we say an ordered tuple of a subset of these variables is in $N_{{\boldsymbol{r}}}$ if the relevant inequalities apply to this subset. For example, $(\kappa,\xi)\in N_{{\boldsymbol{r}}}$ means $|\kappa-1|<r^{(\kappa)}$ and $\frac{|\xi|}{\sigma^{*}}<r^{(\xi)}$. The following will be a local strong-convexity parameter

[TABLE]

Finally, we define one more set of Lipschitz constants. Let

[TABLE]

and let

[TABLE]

Step 6: Lipschitz continuity and strong convexity properties. Using the parameters we defined in (B.37) and (B.38), we let ${\boldsymbol{r}}_{i}=(r^{(\Theta)}_{i},r^{(B)}_{i},r^{(G)}_{i},r^{(\sigma)}_{i},r^{(\kappa)}_{i},r^{(\xi)}_{i})$. Our next lemma establishes the regularity properties we will need.

Lemma B.3**.**

If $(\Theta,B,G,\sigma,\kappa,\xi)\in N_{{\boldsymbol{r}}_{1}}$, then

1. (i)

($\tau$* is locally Lipchitz)*

[TABLE] 2. (ii)

($g$* is locally Lipschitz)*

[TABLE]

If $(\Theta,B,G,\sigma,\kappa,\xi)\in N_{{\boldsymbol{r}}_{2}}$, then

1. (iii)

($\tau$* and $g$ are locally bounded)*

[TABLE] 2. (iv)

(The subgradient at $B^{*}$ is locally Lipshitz) There exists D\in\partial{\mathcal{L}}(B;\Theta,G,\sigma,1,0)\Big{|}_{B=B^{*}} such that

[TABLE] 3. (v)

(The objective is locally strongly convex in $B$)

[TABLE] 4. (vi)

(${\mathcal{L}}$* is locally Lipschitz in $\Theta,G,\sigma,\kappa,\xi$)*

[TABLE]

If $(\Theta,G,\sigma,\kappa,\xi)\in N_{{\boldsymbol{r}}_{1}}$ and $d_{2}\left(\frac{B}{\tau^{*}},\frac{\Theta^{*}}{\tau^{*}}\right)<\frac{4R_{1}}{3\tau^{*}}$, then

1. (vii)

(${\mathcal{L}}$* locally Lipschitz in $\kappa,\xi$ in expanded $B$ neighborhood)*

[TABLE]

Proof of Lemma B.4.

We prove each result in order.

1. (i)

First, we consider $(\Theta,B,G,\sigma,\kappa,\xi)\in N_{{\boldsymbol{r}}_{1}}$. On this neighborhood, we have

[TABLE]

because $\left|\frac{\sigma}{\sigma^{*}}-1\right|<\frac{1}{2}$ by (B.37), (B.39). Also

[TABLE]

by (B.33), (B.37), (B.39), and the triangle inequality. Then

[TABLE]

The first two lines are bounded by

[TABLE]

because the derivative of $\tau$ with respect to $\Theta$ or $B$ on $N_{{\boldsymbol{r}}_{1}}$ has $\ell_{2}$ norm bounded by $\frac{d_{2}(\Theta,B)}{\sigma^{*}\delta/2}<\frac{4R_{1}}{\sigma^{*}\delta}$ by (B.28), (B.50), and (B.51), and we may substitute the Lipschitz constants by (B.34). The third line is bounded by

[TABLE]

because the derivative of $\tau$ with respect to $\sigma$ is bounded by 1 by (B.29) (because $\tau(B,\Theta,\sigma,\kappa,\xi)>\sigma$ always), and we may substitute the Lipschitz constants by (B.34). The fourth line is bounded by

[TABLE]

because the derivative of $\tau$ with respect to $\kappa$ is bounded by

[TABLE]

by (B.30), (B.50), and (B.51), and we may substitute the Lipschitz constants by (B.34). Finally, the fifth line is bounded by

[TABLE]

because the derivative of $\tau$ with respect to $\xi$ is bounded in absolute value by

[TABLE]

by (B.27), (B.31), (B.37), (B.39), and (B.51), and we may substitute the Lipschitz constants by (B.34). Summing the preceding five bounds, we conclude (i). 2. (ii)

By (B.37) and (B.39) that on $N_{{\boldsymbol{r}}_{1}}$ we have $\|G\|_{2}\leq 2$, whence by (B.51), we have

[TABLE]

Combined with (B.35) and (i), we get (ii). 3. (iii)

Observe that $(B,\Theta,G,\sigma,\kappa,\xi)\in N_{{\boldsymbol{r}}_{2}}$ implies $(B,\Theta,G,\sigma,\kappa,\xi)\in N_{{\boldsymbol{r}}_{1}}$ because $r_{2}^{(A)}\leq r_{1}^{(A)}$ for $A\in\{\Theta,B,G,\sigma,\kappa,\xi\}$. We may combine (i) and (ii) with (B.38) to get (B.45). 4. (iv)

By (B.37) and (B.39), on $N_{{\boldsymbol{r}}_{1}}\subset N_{{\boldsymbol{r}}_{2}}$ we have $\|G\|_{2}\leq 2$. Thus,

[TABLE]

Also, by (B.45) both $g$ and $\tau$ are positive on $N_{{\boldsymbol{r}}_{2}}$, so that

[TABLE]

Denoting $g^{*}=g(B^{*};\Theta^{*},G^{*},\sigma^{*},1,0)$ and recalling $\tau^{*}=\tau(B^{*};\Theta^{*},G^{*},\sigma^{*},1,0)$, observe that

[TABLE]

where in the first inequality we have used the triangle inequality; in the first line of the second inequality we have used (B.33) and (ii); in the second line of the second inequality we have used (B.45), (B.33), (i), and the fact that the derivative of $\tau\mapsto 1/\tau$ is $1/\tau^{2}$ which is bounded by $4/{\tau^{*}}^{2}$ on $N_{{\boldsymbol{r}}^{2}}$; in the third line of the second inequality we have used (B.45) and (i); and in the last equality we have used (B.36). Because 0\in\partial{\mathcal{L}}(B;\Theta^{*},G^{*},{\sigma^{*}}^{2},1,0)\Big{|}_{B=B^{*}} and the only part of the objective ${\mathcal{L}}$ which depends upon $\Theta,G,\sigma$ is given by $\frac{1}{2}g_{+}^{2}$, we conclude (B.46). 5. (v)

Combining Lemma B.1, (B.45), and (B.40), we conclude (B.47). 6. (vi)

Because the derivative of $x\mapsto\frac{1}{2}x^{2}$ is $x$, we have by (ii), (B.45), the chain rule, and the fact that the only part of ${\mathcal{L}}$ which depends upon $\Theta,G,\sigma,\kappa,\xi$ is $\frac{1}{2}g_{+}^{2}$, that (vi). 7. (vii)

Because $\left|\frac{\sigma}{\sigma^{*}}-1\right|<r_{1}^{(\sigma)}=\frac{1}{2}$, $|\kappa-1|<r_{1}^{(\kappa)}$, $d_{2}\left(\frac{B}{\tau^{*}},\frac{\Theta}{\tau^{*}}\right)<d_{2}\left(\frac{B}{\tau^{*}},\frac{\Theta^{*}}{\tau^{*}}\right)+d_{2}\left(\frac{\Theta^{*}}{\tau^{*}},\frac{\Theta}{\tau^{*}}\right)<\frac{4R_{1}}{3\tau^{*}}+r_{1}^{(\Theta)}<\frac{2R_{1}}{\tau^{*}}$, $\|G\|_{2}<1+r_{1}^{(G)}=2$, and $\frac{|\xi|}{\sigma^{*}}<r_{1}^{(\xi)}=\frac{\sigma^{*}\sqrt{\delta}}{16R_{1}}$, we have by (B.14) that

[TABLE]

where in the last inequality we apply (B.41). Because the derivative of $x\mapsto\frac{1}{2}x^{2}$ is $x$, we get by (ii)

[TABLE]

where in the last line we have used (B.42).

The proof is complete. ∎

Step 7: Locally stable solution to Gordon’s $L_{2}$ optimization. We return to studying Gordon’s $L_{2}$ optimization (B.15). By viewing it as a perturbation of Gordon’s $L_{2}$ population optimization (Step 2), we can bound the values of ${\mathcal{L}}$ close to and far from $B^{*}$.

Lemma B.4**.**

There exists $c_{1}^{\prime},c_{2}^{\prime},c_{3}^{\prime},L>0$ depending only on $\tau^{*},\lambda^{*},\delta$ such that for

[TABLE]

*if *

[TABLE]

then

[TABLE]

and

[TABLE]

In fact, the constants $c_{1}^{\prime},c_{2}^{\prime},c_{3}^{\prime},L$ are explicitly

[TABLE]

Note that with these choices, we have that whenever $\epsilon<c_{1}^{\prime}$

[TABLE]

because $c_{1}^{\prime}\leq\frac{Kr_{\mathrm{min}}^{2}}{60L}$.

Proof of Lemma B.4.

Note that because $\epsilon<c_{1}^{\prime}\leq r_{\mathrm{min}}$, Eq. (B.65) implies $(B^{*},\Theta,G,\sigma,\kappa,\xi)\in N_{{\boldsymbol{r}}_{2}}$, whence we have by (vi) and substituting (B.21) that (B.66) holds.

Moreover, for $(B,\Theta,G,\sigma,\kappa,\xi)\in N_{{\boldsymbol{r}}_{2}}$, we have by (B.46) and (B.47) that there exists D\in\partial{\mathcal{L}}(B;\Theta,G,\sigma,1,0)\Big{|}_{B=B^{*}} such that

[TABLE]

where in the second inequality we have applied the definition of $L$ (B.68) and Cauchy-Schwartz. By (B.69), we may apply the previous display to all $B$ with $d_{2}\left(\frac{B}{\tau^{*}},\frac{B^{*}}{\tau^{*}}\right)=c_{3}^{\prime}\sqrt{\epsilon}$, whence for such $B$

[TABLE]

where the last inequality holds because $3L\epsilon\sqrt{\frac{60L}{K}\epsilon}<15L\epsilon$ because $\epsilon<c_{1}^{\prime}<\frac{5K}{12L}$. By convexity in $B$ (which holds because $\xi=0$), Eq. (B.71) holds also for any $B$ with $d_{2}\left(\frac{B}{\tau^{*}},\frac{B^{*}}{\tau^{*}}\right)\geq c_{3}^{\prime}\sqrt{\epsilon}$.

Now consider any $(\Theta,G,\sigma,\kappa,\xi)$ satisfying (B.65) and $B$ satisfying both $c_{2}^{\prime}\geq d_{2}\left(\frac{B}{\tau^{*}},\frac{\Theta^{*}}{\tau^{*}}\right)$ and $d_{2}\left(\frac{B}{\tau^{*}},\frac{B^{*}}{\tau^{*}}\right)\geq c_{3}^{\prime}\sqrt{\epsilon}$. Then

[TABLE]

where in the first inequality we have applied (B.49), which is permitted because $d_{2}\left(\frac{B}{\tau^{*}},\frac{\Theta^{*}}{\tau^{*}}\right)\leq c_{2}^{\prime}=\frac{4R_{1}}{3\tau^{*}}$, and (vi), which is permitted because $(B^{*},\Theta,G,\sigma,\kappa,\xi)\in N_{{\boldsymbol{r}}_{2}}$; in the second inequality we have used the definition of $L$ in (B.68); and in the third inequality we have used (B.71), which is permited becuase $d_{2}\left(\frac{B}{\tau^{*}},\frac{B^{*}}{\tau^{*}}\right)\geq c_{3}^{\prime}\sqrt{\epsilon}$. Substituting (B.21) yields (B.67). ∎

Step 7: Control of Gordon’s $\mathbb{R}^{p}$ perturbed optimization. We complete the proof of Lemma 4.3 by viewing the objective (4.13) as a restricted version of the objective ${\mathcal{L}}$ and using the control over the latter objective established in Lemma B.4. Throughout this section, we take $L,r_{\mathrm{min}},c_{1}^{\prime},c_{2}^{\prime},c_{3}^{\prime}$ as defined by (B.68).

Let $\kappa=\frac{\|{\boldsymbol{h}}\|^{2}}{n}$, $\sigma^{2}=\frac{\|{\boldsymbol{w}}\|^{2}}{n}$, $\xi=\left\langle\frac{{\boldsymbol{h}}}{\sqrt{n}},\frac{{\boldsymbol{w}}}{\sqrt{n}}\right\rangle$, and $\delta=n/p$. We may then write

[TABLE]

Let ${\boldsymbol{b}}={\boldsymbol{\theta}}+{\boldsymbol{v}}$, and we denote a reparametrized version of the objective

[TABLE]

By Lemma 3.4.(a), there exists $\Theta^{*},G^{*},\Theta,G\in L_{2}(0,1)$ such that $\Theta^{*}\sim\mu_{{\boldsymbol{\theta}}}$, $G^{*}\sim{\sf N}(0,1)$ independent, $(\Theta,\tau^{*}G)\sim\mu_{({\boldsymbol{\theta}},\tau{\boldsymbol{g}})}$, and $d_{2}\left(\frac{\Theta}{\tau^{*}},\frac{\Theta^{*}}{\tau^{*}}\right)^{2}+d_{2}(G,G^{*})^{2}=W_{2}(\mu_{({\boldsymbol{\theta}}/\tau^{*},{\boldsymbol{g}})},\mu_{{\boldsymbol{\theta}}/\tau^{*}}\otimes{\sf N}(0,1))^{2}$. Because ${\boldsymbol{\theta}},{\boldsymbol{g}}\in\mathbb{R}^{p}$, we have that $\Theta,G$ are measurable with respect to the $\sigma$-algebra generated by a partition consisting of $p$ atomic sets each with probability $1/p$. We denote this $\sigma$-algebra by ${\mathcal{I}}_{p}$, and we denote the ${\mathcal{I}}_{p}$-measurable random variables in $L_{2}(0,1)$ by $m{\mathcal{I}}_{p}$. Let $\iota:\mathbb{R}^{p}\rightarrow L_{2}(0,1)$ be the linear isomorphism of the form (3.18) between $\mathbb{R}^{p}$ and $m{\mathcal{I}}_{p}$ which takes ${\boldsymbol{\theta}}$ to $\Theta$ and ${\boldsymbol{g}}$ to $G$. Observe that for all ${\boldsymbol{b}}$,

[TABLE]

where we have used that $f_{p}({\boldsymbol{b}})=f(\iota(B))$ because $f$ is an $L_{2}$ embedding of $pf_{p}$ (see (3.20)).

Denote $B^{*}=\eta\circ(\Theta^{*}+\tau^{*}G^{*})$. With $c_{1}^{\prime}$ as in Lemma B.4, assume there exists $0<\epsilon<c_{1}^{\prime}$ such that

[TABLE]

In particular, then $d_{2}\left(\frac{\Theta}{\tau^{*}},\frac{\Theta^{*}}{\tau^{*}}\right)<\epsilon$ and $d_{2}(G,G^{*})<\epsilon$. By Lemma B.4, Eq. (B.66), we have

[TABLE]

By Lemma B.3 and B.4, the objective ${\mathcal{L}}(\cdot;\Theta,G,\sigma,1,0)$ is strongly convex and lower semi-continuous on $d_{2}\left(\frac{B}{\tau^{*}},\frac{B^{*}}{\tau^{*}}\right)<c_{3}^{\prime}\sqrt{\epsilon}$, a set outside of which it is uniformly sub-optimal. Thus, its minimum is uniquely achieved at some $B_{\mathrm{opt}}$ with $d_{2}\left(\frac{B_{\mathrm{opt}}}{\tau^{*}},\frac{B^{*}}{\tau^{*}}\right)<c_{3}^{\prime}\sqrt{\epsilon}<c_{1}^{\prime}\leq r_{\mathrm{min}}\leq r_{1}^{(B)}=\frac{R_{1}}{3\tau^{*}}$, where we have used (B.69) in the second inequality. Combining this with (B.33), we get $d_{2}\left(\frac{B_{\mathrm{opt}}}{\tau^{*}},\frac{\Theta^{*}}{\tau^{*}}\right)<\frac{R_{1}}{3\tau^{*}}+\frac{R_{1}}{\tau^{*}}=c_{2}^{\prime}$. Then, because by (B.75) also $d_{2}\left(\frac{\Theta}{\tau^{*}},\frac{\Theta^{*}}{\tau^{*}}\right)<\epsilon<c_{1}^{\prime}\leq r_{\mathrm{min}}\leq r_{2}^{(\Theta)}$ and similarly $\left|\frac{\sigma}{\sigma^{*}}-1\right|<r_{2}^{(\sigma)}$, $|\kappa-1|<r_{2}^{(\kappa)}$, and $\frac{|\xi|}{\sigma^{*}}<r_{2}^{(\xi)}$, by (B.49),

[TABLE]

where the second inequality holds by (B.76) and optimality. By Lemma B.2, $B_{\mathrm{opt}}$ is $\sigma(\Theta+\tau G)$ measurable, whence

[TABLE]

Next, let

[TABLE]

Then

[TABLE]

where the first inequality holds because $\iota(A)$ is contained in the set over which the infimum in the second line is taken, and the second inequality holds by Lemma B.4.

By [MM18, Proposition F.2], there exist $\mathsf{C_{conc}},\mathsf{c_{conc}}:\mathbb{R}_{>0}\rightarrow\mathbb{R}_{>0}$, non-increasing and non-decreasing respectively, such that for $0<\epsilon<1/2$,

[TABLE]

where $C=\mathsf{C_{conc}}(\mathsf{snr})$, $c=\mathsf{C_{conc}}(\mathsf{snr})$, and $\mathsf{snr}=\|{\boldsymbol{\theta}}\|^{2}/(p\tau^{2})$. Now note that ${\mathbb{E}}[\exp(\lambda(h_{i}^{2}-1))]=\frac{e^{-\lambda}}{\sqrt{1-2\lambda}}\leq e^{-\lambda}e^{\lambda+4\lambda^{2}}=e^{4\lambda^{2}}$ for $\lambda\leq 1/4$ because the second derivative of $\log(1-2\lambda)^{-1/2}$ is bounded in absolute by 8 on $[-1/4,1/4]$. Thus, taking $\lambda=\epsilon/8\in[-1/4,1/4]$ for $0<\epsilon<1/2$, we have

[TABLE]

By standard Gaussian concentration

[TABLE]

Because (B.75) implies (B.79) and (B.81), we conclude that for $\epsilon<\min\left\{c_{1}^{\prime},\frac{1}{2}\right\}$, we have

[TABLE]

where

[TABLE]

where we have used the monotonicity of $\mathsf{C_{conc}}$ and $\mathsf{c_{conc}}$, replaced ${\boldsymbol{b}}$ with ${\boldsymbol{\theta}}+{\boldsymbol{v}}$, and used (4.19). Now (4.3) follows after the change of variables $c_{3}^{\prime 2}\epsilon\rightarrow\epsilon$ with the functions

[TABLE]

where the dependence of all parameters which appear in this display on $M,\tau^{*},\sigma^{*},\delta$ is specified in Step 5, (B.68), and (B.85). Note that the choice of $\mathsf{c_{2}}$ is made to guarantee that for all $0<\epsilon<1/2$ we have

[TABLE]

Appendix C Additional simulations

We now present additional simulation results which demonstrate our results empirically. We consider three penalties.

Power of $\ell_{2}$-norm. First we consider the penalty

[TABLE]

where $\alpha\geq 1$. When $\alpha=2$, this penalty is separable. For this choice of exponent, using this penalty in linear regression implements ridge regression. In the sequence model, the separable proximal operator (1.3) applies a linear estimator to each coordinate with a slope which is constant across coordinates and does not depend upon the distribution of the data. For choices of $\alpha\neq 2$, a linear estimator is applied to each coordinate with a slope which is constant across coordinates but which does depend upon the distribution of the data. To demonstrate this fact, we plot in the first row of Figure 2 theory and simulation results for three choices of expoenent $\alpha=1,2$, and $4$. We take $p=1000$ and consider estimating ${\boldsymbol{\theta}}\in\mathbb{R}^{1000}$ whose $j^{\text{th}}$ entry is set to the $j/(p+1)^{\text{st}}$ quantile of the $\mathsf{N}(0,1)$ distribution. For each of $\tau=.25,1$, and $5$, we once generate observations ${\boldsymbol{y}}={\boldsymbol{\theta}}+\tau{\boldsymbol{z}}$ for ${\boldsymbol{z}}\sim\mathsf{N}(0,I_{p})$ and compute $\widehat{\boldsymbol{\theta}}$ as in (1.3) in $\mathsf{R}$ using the $\mathsf{CVXR}$ package. We plot the estimated value $\widehat{\theta}_{j}$ against the observation $y_{j}$ for 100 randomly sampled coordinates across the three values of $\tau$. We also plot theory curves of ${\mathfrak{A}}_{f_{p}}(\mu_{{\boldsymbol{\theta}}}^{*\tau})$. The agreement between theory and experiment is extremely good. We see that for $\alpha=2$, ${\mathfrak{A}}_{f_{p}}(\mu_{{\boldsymbol{\theta}}}^{*\tau})$ does not depend upon $\tau$, as we expect for separable, and hence non-adaptive, penalties. For $\alpha=1$, we see that shrinkage decreases with noise-level and for $\alpha=4$ shrinkage increases with noise-level. In this context, we want greater shrinkage with higher-noise level, suggesting $\alpha>2$ is a good choice.

Power of $\ell_{1}$ norm. Next we consider the penalty

[TABLE]

where $\alpha\geq 1$. When $\alpha=1$, this penalty is separable. For this choice of exponent, using this penalty in linear regression implements the LASSO. In the sequence model, the separable proximal operator (1.3) applies soft-thresholding to each coordinate with a threshold which is constant across coordiantes and does not depend upon the distribution of the data. For choices $\alpha>1$, soft thresholding is applied to each coordinate with a threshold which is constant across coordinates but which does depend upon the distribution of the data. In the second row of Figure 2, we plot theory and experiments for $p=1000$ as above. Now, we take ${\boldsymbol{\theta}}$ with has 50 coodinates equal to ${-1}$, 50 coordinates equal to 1, and the remaining 900 coordinates equal to 0. We plot the estimated value $\widehat{\theta}_{j}$ against the observation $y_{j}$ for 100 randomly sampled coordinates across the three values of $\tau$. We also plot theory curves of ${\mathfrak{A}}_{f_{p}}(\mu_{{\boldsymbol{\theta}}}^{*\tau})$. We see that the threshold increases with noise-level, as it should. All computations are done in $\mathsf{R}$ with pacakge $\mathsf{CVXR}$.

Smoothed ordered weighted $\ell_{1}$-norms. Finally, we display some additional results for the penalty (1.7) discussed in Secton (1.7).

In the third row of Figure 2, we plot theory and experiments for $p=1000$. The left plot repeats Figure 1, and we refer the reader to Section 1.1 for a description of that plot. The right plot displays the effect of varying $M$ instead of $\tau$. For a positive number $M$, we take ${\boldsymbol{\theta}}$ with 50 coordinates equal to $-M$, 50 coordinates equal to $M$, and 900 coordinates equal to [math]. We consider three settings of the signal strength $M=0,1,10$ at noise level $\tau=1$.