On the support recovery of marginal regression

TL;DR

This paper analyzes the support recovery capabilities of marginal regression in high-dimensional settings, providing a detailed characterization of its performance and factors affecting its success, especially under correlated design conditions.

Contribution

It offers a comprehensive analysis of marginal regression's support recovery, introducing the concept of MR incoherence and comparing it with other feature selection methods.

Findings

Identifies MR incoherence as key to support recovery performance.

Provides a nuanced characterization of MR's effectiveness.

Relates MR behavior to methods like Lasso, OMP, and SIS.

Abstract

Leading methods for support recovery in high-dimensional regression, such as Lasso, have been well-studied and their limitations in the context of correlated design have been characterized with precise incoherence conditions. In this work, we present a similar treatment of selection consistency for marginal regression (MR), a computationally efficient family of methods with connections to decision trees. Selection based on marginal regression is also referred to as covariate screening or independence screening and is a popular approach in applied work, especially in ultra high-dimensional settings. We identify the underlying factors---which we denote as \emph{MR incoherence}---affecting MR's support recovery performance. Our near complete characterization provides a much more nuanced and optimistic view of MR in comparison to previous works. To ground our results, we provide a broad taxonomy of results for leading feature selection methods, relating the behavior of Lasso, OMP, SIS, and MR. We also lay the foundation for interesting generalizations of our analysis, e.g., to non-linear feature selection methods and to more general regression frameworks such as a general additive models.

Equations

$$\begin{array}{rl}Y={\beta }^{T}X+\epsilon ,\text{where}\mathbb{E}(\epsilon )& =0,\mathrm{var}(\epsilon )={\sigma }^2,\\ \mathbb{E}(X)& =0_p,\mathrm{cov}(X)=\Sigma ,{\Sigma }_{ii}=1,\forall i\in [p],\\ \mathrm{cov}(X,\epsilon )& =0_p\end{array}$$

$$\displaystyle\Gamma_{S}:=\Gamma_{S,\rho,R}:=\big{\{}\beta\in\mathbb{R}^{p}\;|\;\beta_{S^{c}}=0,\;\;|\beta_{S}|_{\min}>\rho,\;\;\|\beta_{S}\|_{\infty}\leq R|\beta_{S}|_{\min}\big{\}}$$

$$\mathcal{A}({\mathbb{P}}_{\beta ,\Sigma })=\mathrm{supp}(\beta ),\forall {\mathbb{P}}_{\beta ,\Sigma }\in {\mathcal{P}}_{\text{lin}}^{p+1},\beta \in {\Gamma }_S.$$

$$\mathcal{A}({\mathbb{P}}_{\beta ,{\Sigma }^{\prime }})=\mathrm{supp}(\beta ),\forall {\mathbb{P}}_{\beta ,{\Sigma }^{\prime }}\in {\mathcal{P}}_{\text{lin}}^{p+1},\beta \in {\Gamma }_S,{\Sigma }^{\prime }\in B_{\infty }(\Sigma ,\beta ;\delta /2)$$

$$\frac{R}{1+R}\Vert {\Sigma }_{Sj}\pm {\Sigma }_{Sk}{\Vert }_1+\frac{{\delta }^{\prime }}{(1+R)}<{\Sigma }_{jj}\pm {\Sigma }_{jk},\forall j\in S,k\in S^c.$$

$$\displaystyle\Sigma\in\operatorname{MRI}_{S}\Big{(}\frac{\delta}{\rho};R\Big{)}$$

$$\displaystyle\|\Sigma_{Sk}\|_{1}<\frac{1}{R}\Big{(}1-\delta^{\prime}\Big{)}+\Big{(}1-\frac{1}{R}\Big{)}|\Sigma_{Sk}|_{\min},\quad\forall k\in S^{c}.$$

$$|{\Sigma }_{jj}\pm {\Sigma }_{jk}|\le {\lambda }_s\sqrt{s}+\frac{1}{2}\Vert {\Sigma }_{Sj}\pm {\Sigma }_{Sk}{\Vert }_1.$$

$$\underset{k\in S^c}{\max }\Vert {\Sigma }_{SS}^{-1}{\Sigma }_{Sk}{\Vert }_1=\frac{\eta (r+\mu (3r-4))}{(1-\mu )(1-\mu +r\mu )}\le 1$$

$$\displaystyle|\eta|<\min\Big{\{}\frac{-\mu+\zeta}{1-\zeta},\frac{\mu+\zeta}{1+\zeta}\Big{\}},\quad|\mu|<\zeta:=\frac{1}{R(r-1)}.$$

$$\hat{r}=({\hat{r}}_j)=\frac{1}{n}\sum _{i=1}^n{x}_i{y}_i\in {\mathbb{R}}^p.$$

$$\xi (\Sigma ;\Gamma ,t):=\underset{\beta \in \Gamma ,\Vert \beta {\Vert }_2\le t}{\sup }\sqrt{{\beta }^T\Sigma \beta }=\underset{\beta \in \Gamma ,\Vert \beta {\Vert }_2\le t}{\sup }\Vert {\Sigma }^{1/2}\beta {\Vert }_2.$$

$$\displaystyle\|\widehat{r}-r\|_{\infty}\;\leq\;\big{(}\xi(\Sigma;\Gamma_{S},t)+\sigma\big{)}\sqrt{{c_{2}\log p}/{n}}.$$

$$\displaystyle\Sigma\,\in\,\operatorname{MRI}_{S}\Big{(}2\big{(}\xi(\Sigma;\Gamma_{S},t)+\sigma\big{)}\frac{1}{\rho}\sqrt{{c_{2}\log p}/{n}};\,R\Big{)}.$$

$${\beta }^T\Sigma \beta ={\beta }_S^T{\Sigma }_{SS}{\beta }_S=\Vert {\Sigma }_{SS}^{1/2}{\beta }_S{\Vert }_2^2\le |||{\Sigma }_{SS}^{1/2}||{|}_{\text{2}}^2\Vert {\beta }_S{\Vert }_2^2=|||{\Sigma }_{SS}||{|}_{\text{2}}\Vert \beta {\Vert }_2^2,$$

$$\displaystyle\Sigma\,\in\,\operatorname{MRI}_{S}(\delta;\,R),\quad\text{and}\quad n\,\,\gtrsim\,\,\delta^{-2}\big{(}\sigma^{2}+|\!|\!|\Sigma_{SS}|\!|\!|_{\text{2}}\min\{t^{2},s\rho^{2}R^{2}\}\big{)}\,\rho^{-2}\log p.$$

$${\delta }_{2s}(\Sigma )<\frac{(1-{\delta }^{\prime })}{R}\frac{2s-1}{s-1}.$$

$$\displaystyle\mathcal{F}:=\bigcap_{j,k\in S^{c},\;\lambda\in[-1,1]}\big{\{}f\in\mathcal{F}_{0}:\;|\langle Y_{j,k}^{\lambda},f^{\prime}\rangle_{L^{2}}|\;\geq\;\alpha\|Y_{j,k}^{\lambda}\|_{L^{2}}\,\|f^{\prime}\|_{L^{2}}\big{\}}$$

$$|||{\Sigma }_{S^{c}S}{\Sigma }_{SS}^{-1}||{|}_{\infty }\le 1-\delta$$

$$\tilde{\beta }\in \underset{\beta \in {\mathbb{R}}^p}{\mathrm{argmin}}\frac{1}{2}\mathbb{E}(Y-{\beta }^{T}X{)}^2+\lambda \Vert \beta {\Vert }_1$$

$$|⟨{\Sigma }_{\ast k},\beta ⟩|+\delta <|⟨{\Sigma }_{\ast j},\beta ⟩|,\forall \beta \in {\Gamma }_S,j\in S,k\in S^c.$$

$$\displaystyle\Gamma^{\dagger}=\big{\{}\phi\in\mathbb{R}^{p}:\;|\langle\phi,\beta\rangle|>\delta,\;\forall\beta\in\Gamma\big{\}}.$$

$${\Sigma }_{\ast j}+\lambda {\Sigma }_{\ast k}\in {\Gamma }_S^{†},\text{for all}(j,k)\in S\times S^c,\text{and}\lambda \in [-1,1].$$

$${\Gamma }_S^{\prime }$$

$${\Omega }_S\subseteq {\Gamma }_S^{\prime \prime }$$

$$\frac{R}{1+R}\Vert {\Sigma }_{Sj}+\lambda {\Sigma }_{Sk}{\Vert }_1+\frac{{\delta }^{\prime }}{1+R}<\Vert {\Sigma }_{Sj}+\lambda {\Sigma }_{Sk}{\Vert }_{\infty },\forall j\in S,k\in S^c,\lambda \in [-1,1].$$

$$\frac{R}{1+R}\Vert {\Sigma }_{Sj}+\lambda {\Sigma }_{Sk}{\Vert }_1+\frac{{\delta }^{\prime }}{1+R}<{\Sigma }_{jj}+\lambda {\Sigma }_{jk},\forall j\in S,k\in S^c,\lambda \in [-1,1].$$

$${\alpha }_R(\varphi )=\underset{\beta \in {\Gamma }_{S,R}}{\min }|⟨\varphi ,\beta ⟩|$$

$$⟨\varphi ,\beta ⟩=\sum _{j\in S_1}|{\varphi }_j|-\sum _{j\in S_0}|{\varphi }_j|$$

$$\alpha (\varphi )=\sum _{j\in S_1(\varphi )}|{\varphi }_j|-\sum _{j\in S_0(\varphi )}|{\varphi }_j|\ge 0.$$

$${\beta }_{j^{\ast }}=\mathrm{sign}({\Sigma }_{j^{\ast }k})\rho ,{\beta }_{j^{\prime }}=\mathrm{sign}({\Sigma }_{j^{\prime }k})R\rho ,\forall j^{\prime }\in S\backslash \{j^{\ast }\}.$$

Full text

Keywords: Support recovery; high-dimensional regression; marginal regression; independence screeing; covariate screeing; incoherence condition

1 Introduction

Support recovery in high-dimensional regression is a well-studied problem, and of significant practical importance, e.g., in the context of model interpretability. Leading methods such as Lasso and other $\ell_{1}$-regularization variants [Hastie et al., 2015] have computational complexity $O(p^{2}n)$ which introduces significant computational overhead for large $p$. These methods demonstrate limitations for support recovery when correlation in the design exceeds moderate amounts, as characterized by Lasso incoherence.

We focus on an alternative approach, marginal regression (MR), which is attractive for its algorithmic simplicity, computational efficiency, and its capability for embarrassing parallelism. In particular, MR independently compares each covariate to the response, following a procedure that closely resembles the splitting criterion used for decision trees [Kazemitabar et al., 2017]. The greedy nature of this approach suggests that it may be subject to significantly more onerous limitations on correlation in the design. It is worth noting that selection based on marginal regression goes by many other names such covariate screening or independence screening, and is a popular approach in applied work, especially in ultra high-dimensional settings.

Our primary contribution in this paper is showing that the conditions on MR support recovery are not nearly as pessimistic as have been previously assumed. In particular, while pairwise incoherence (PWI) is known to be a sufficient condition [Donoho and Huo, 2001, Donoho and Elad, 2003], we show that it is overly stringent. Our results demonstrate that the behavior of MR is much more nuanced than what PWI predicts. We establish this claim by providing a near complete characterization of the support recovery performance of MR, revealing the role of various parameters and drawing some surprising conclusions about the strength of MR in certain situations (and its weaknesses in others).

More precisely, we derive a condition on the covariance matrix $\Sigma\in\mathbb{R}^{p\times p}$ of the variables which we call MR incoherence. We also introduce a parameter $R\in[1,\infty]$ that controls the spread of the regression coefficients $\beta$, that is: $|\beta_{i}|\leq R|\beta_{j}|$ for all $i,j\in[p]$. We show that MR incoherence is necessary and sufficient for recovery when $R\geq 2$ and sufficient when $R\in[1,2)$. Our results also hinge on the correlation among the on-support variables, namely the $\Sigma_{SS}$ block of the covariance matrix $\Sigma$—where $S$ is the support of $\beta$. There are several remarkable consequences of our analysis:

• •

MR can benefit from sparsity of the covariance structure, i.e., if the on-support covariates correlate in pairs (together with a low spread of the coefficient parameters), MR has a comparable performance to that of the Lasso. More generally, when on-support covariates correlate in groups of size $r$, MR shows a substantial deviation in performance from PWI and compete closer to that of Lasso.

• •

When the on-support covariates are nearly uncorrelated and the regression coefficients are not spread out, the support recovery performance of MR again approaches that of the Lasso; this fact is independent of the correlation between off-support and on-support variables.

• •

MR incoherence implies restricted isometry property (RIP), a known sufficient condition for Lasso.

• •

The uniform performance of MR is sensitive to the minimum eigenvalue of the covariance matrix, which contradicts the intuition from prior work that does not consider uniform recovery within a class of parameters [Genovese et al., 2012].

We owe these results to our novel approach which incorporates the spread of the regression coefficients. Prior work either neglected it as a factor [Genovese et al., 2012] or failed to separate its effect from the covariance matrix [Fan and Lv, 2008], which led to rejecting the possibility of uniform recovery for MR [Robins et al., 2003]. Our novel approach lays the foundation for interesting generalizations of our analysis. In Section 3, we leverage our results to present a broad taxonomy of the necessary and sufficient conditions for a wide range of feature selection methods, and relate the performance of marginal regression with popular feature selection methods including Lasso, (Orthogonal) Matching Pursuit, and SIS. Moreover, as we discuss in Section 4, it is possible to extend our results to non-linear feature selection methods such as Dstump [Kazemitabar et al., 2017] or information gain, and to more general regression frameworks such as a general additive model under suitable regularity conditions.

Related work.

In this work we study uniform recovery by MR, as opposed to previous works which studied the average case and “a fixed single parameter” cases. We precisely characterize how the spread of regression coefficients and the structure of the on-support covariance plays a role in the selection consistency of MR. The necessary and sufficient conditions we provide are new to the best of our knowledge and are more relaxed as well as clearer than earlier results.

That a form of PWI is sufficient for MR support recovery, in the fixed design regression setting, is colloquially known and often attributed to the work by Donoho and Huo [2001] and Donoho and Elad [2003], although we could not find this exact result there or in the literature. Theorem 1, which gives the necessary and sufficient conditions for recovery in terms of MR incoherence (cf. Definition 1) clearly shows that conditions needed for MR recovery are in general weaker than PWI. Moreover, if $R=\infty$, no amount of PWI can be tolerated by MR, countering the colloquial knowledge.

Genovese et al. [2012] showed that uniform recovery by MR is not possible when $R$ is unbounded (except in the trivial case of $\Sigma=I$). This is the content of Lemma 2 in our work. Due to this negative result, the bulk of the work in Genovese et al. [2012] focused on average case recovery (i.e., putting a sparse prior on $\beta$, the regression coefficient vector, and recovering the support with high probability). In contrast, we show that it is possible to recover uniformly over a class of coefficients, assuming $R<\infty$.

Fan and Lv [2008] is among the earliest and most noted work on MR. They termed the approach sure independence screening (SIS) and provided sufficient asymptotic guarantees, as the sample size $n\to\infty$, for SIS to recover a superset guaranteed to contain the true support with high probability. The sufficient conditions in Fan and Lv [2008] are tangled with other assumptions on the sampling process and hence hard to compare with known incoherence-based results.

Finally, we note that by using Pearson correlation for importance scoring, MR can be viewed as a filter method, i.e., a method that independently scores covariates based on their relevance to the target. In contrast, a wide range of (forward) wrapper feature selection methods iteratively evaluate the importance of covariates by including them in the model one after another, with each iterative decision typically based on some importance score. Matching and orthogonal matching pursuit [Donoho and Huo, 2001, Donoho and Elad, 2003] are examples of wrappers that use Pearson correlation, just as in MR, but adjust for the interdependence of the covariates by greedily selecting the most important ones and working with the residuals. Lasso can also be implemented in a similar way, often referred to as forward stagewise regression [Hastie et al., 2007]. In Section 3 we leverage our novel results to connect MR with these wrapper methods, and more broadly to provide a taxonomy of existing and conjectured results, as well as open questions. As part of this discussion, we also relate MR to Sure Independence Screening [Fan and Lv, 2008], a method closely related to MR but solving an easier problem.

Amendment to related work.

After writing this paper, we noticed that an earlier work [Wang et al., 2015] provides results comparable to what we have achieved. The authors in that paper define an incoherence condition, named restricted diagonally dominant (RDD) condition, that is close to our main condition (Definition 1) in form. They show that RDD is a tight guarantee for uniform recovery using marginal regression. Our work deviates from theirs in that they consider signed support recovery (SSR) whereas we consider just the support recovery (SR), i.e., we do not require the sign of coefficients to be recovered correctly. In this sense, our work is complementary to Wang et al. [2015]. Their condition (RDD) is strictly stronger than MRI (that is, RDD $\implies$ MRI) and the extra strength maps to the “sign recovery” part of the problem.

Sign recovery is often assumed as a technical device since it greatly simplify the analysis. Here, we directly derive necessary and sufficient conditions for SR without any sign requirement leading to the MRI conditions. Considering the work of Wang et al. [2015] and ours together reveals an interesting point: Dropping the sign requirement changes the nature of the problem; while RDD is necessary and sufficient for SSR for the whole range of $R>0$ (see (2) for the definition of $R$), MRI is only so for $R\in[2,\infty)$; for $R\in(1,2)$ the necessary and sufficient condition for SR will be combinatorial. That is, there is a dichotomy in the SR problem which is not in SSR. This requires non-elementary arguments in our case as opposed to the short analysis of Wang et al. [2015] (e.g., the sufficiency proof in Wang et al. [2015] does not go through if RDD is replaced with MRI.) We believe our technical contributions here will be of interest since as far as we know, no other necessary and sufficient condition of the SR type is available (even for the Lasso).

Notation.

For any vector $\beta\in\mathbb{R}^{p}$, let us write $|\beta|_{\min}=\min_{j\in[p]}|\beta_{j}|$. For $S\subset[p]$, $\beta_{S}=(\beta_{i},i\in S)$ is the subvector of $\beta$ on indices $S$, and hence $|\beta_{S}|_{\min}=\min_{j\in S}|\beta_{j}|$. Similar notation is used for sub-blocks of matrices, e.g., $\Sigma_{SS^{c}}$ is the block of $\Sigma$ indexed by rows $S$ and columns $S^{c}$. We write $|\!|\!|A|\!|\!|_{p}$ for the $\ell_{p}$ operator norm of a matrix. For example, $|\!|\!|A|\!|\!|_{\infty}$ is the $\ell_{\infty}$ operator norm which is equal to the maximum absolute row sum. Similarly, $|\!|\!|A|\!|\!|_{\text{2}}$ is the usual $\ell_{2}$ operator norm. We use $A_{i*}$ and $A_{*j}$ to denote the $i$th row and the $j$th column of $A$, respectively. The symbols $\lesssim$ and $\gtrsim$ are used to denote inequalities up to constants.

2 Selection by marginal regression

In this section, we first formalize the exact support recovery problem and state mutual incoherence conditions for MR to recover the support uniformly over a controlled class of parameters. We study the problem first at the population level, and then extend the results to the finite sample regime.

2.1 Marginal regression at the population level

At the population level, a random design linear model with response $Y\in\mathbb{R}$, covariate (or feature) vector $X=(X_{1},\dots,X_{p})\in\mathbb{R}^{p}$ and noise $\varepsilon\in\mathbb{R}$, is of the form

[TABLE]

and in addition we assume $\mathbb{E}(X)=0$ and $\operatorname{cov}(X,\varepsilon)=0$, i.e., the noise and the covariate vector are uncorrelated. Note that we are working with a random design model, i.e., $X$ is a random variable. The covariance matrix of $X$, namely, $\Sigma$ will play the prominent role in the support recovery conditions presented here. The diagonal scaling $\Sigma_{ii}=1$ is natural for studying a correlation based approach such as marginal regression and it is inline with common practice of standardizing covariates before performing regression.

Let us fix a subset $S\subset[p]$ with $|S|=s$, which will serve as the true support of $\beta$ to be recovered. We will assume that $s$ is known and available to the algorithms. The class of parameters of interest in this paper is:

[TABLE]

where $R\in[1,\infty]$. Note that the support of any $\beta\in\Gamma_{S}$ is contained in $S$. The parameters $\rho$ and $R$ control, respectively, the so-called minimum signal strength and the spread of the on-support parameters. The two extremes $R=1$ versus $R=\infty$ correspond to equal magnitude for the on-support elements versus no restriction on the relative sizes of $\beta_{j},j\in S$. In other words, for $R=1$, $\beta_{S}$ is a scaled multiple of a sign vector: $\beta_{S}\in\bigcup_{t\geq\rho}\{-t,t\}^{s}$.

The (worst-case) support recovery problem over $\Gamma_{S}$ can be stated as follows: Given that $(Y,X)$ follows model (1) with $\beta\in\Gamma_{S}$, can we recover the support of $\beta$, i.e., the set of indices of its nonzero elements? The guarantee of recovery should hold uniformly over $\beta\in\Gamma_{S}$. To make this notion more precise, let $\mathcal{P}_{\text{lin}}^{p+1}$ be the class of distributions for $(Y,X)$ that satisfy (1). We write $\mathbb{P}_{\beta,\Sigma}\in\mathcal{P}_{\text{lin}}^{p+1}$ for any distribution for $(Y,X)$ that satisfies (1) with regression coefficient $\beta$ and feature covariance $\Sigma$.

A population level support recovery algorithm $\mathcal{A}$ takes a distribution $P_{\beta,\Sigma}$ and outputs a subset of $[p]$ that is believed to be the support of $\beta$. Formally, such an algorithm is a map $\mathcal{A}:\mathcal{P}_{\text{lin}}^{p+1}\to[p]$. We say that $\mathcal{A}$ succeeds in support recovery uniformly over $\Gamma_{S}$ if

[TABLE]

If (3) holds, we also say that $\mathcal{A}$ is model selection consistent over $\Gamma_{S}$ at the population level. Ideally one would like (3) to hold for all nonsingular $\Sigma$ as well. However, for any particular algorithm one might need additional constraints on $\Sigma$ for (3) to hold. These conditions are often called incoherence conditions as they measure various sorts of deviations of $\Sigma$ from the identity. Our goal is to derive incoherence conditions for the marginal regression to succeed in support recovery over $\Gamma_{S}$.

In anticipation of the results under sampling, we also introduce a robust version of (3): $\mathcal{A}$ succeeds in support recovery uniformly over $\Gamma_{S}$ with slack $\delta$ if

[TABLE]

where $\operatorname{\mathbb{B}}_{\infty}(\Sigma,\beta;\delta):=\{\Sigma^{\prime}:\;\|(\Sigma^{\prime}-\Sigma)\beta\|_{\infty}\leq\delta\}$.

The population level marginal regression (MR) performs the following operation: (1) Let $r=(r_{j}):=\operatorname{cov}(X,Y)\in\mathbb{R}^{p}$ and sort the coordinates so that $|r_{i_{1}}|\geq|r_{i_{2}}|\geq\dots\geq|r_{i_{p}}|$. (2) Output $\{i_{1},\dots,i_{s}\}$. The key condition controlling the behavior of population MR is the following:

Definition 1** (MR incoherence).**

A covariance matrix $\Sigma$ satisfies MR incoherence with parameters $R$ and slack $\delta^{\prime}$ relative to subset $S$, denoted as $\Sigma\in\operatorname{MRI}_{S}(\delta^{\prime};R)$, if

[TABLE]

Here, $\pm$ signs go together, that is, (5) represents two sets of inequalities.

As a set of matrices, $\operatorname{MRI}_{S}(\delta^{\prime};R)$ is decreasing in both its argument $\delta^{\prime}$ and $R$. That is, (6) becomes more restrictive as we increase $\delta^{\prime}$ or $R$. It is also worth nothing that $\operatorname{MRI}_{S}(\delta^{\prime};R)$ defines a convex subset of the cone of positive semidefinite matrices. Our main result regarding support recovery performance of the population MR is the following:

Theorem 1** (Population MR consistency).**

Under linear model (1), assume that

[TABLE]

for some $\rho,\delta>0$ and $S\subset[p]$. Then, the following holds:

1. (a)

For any $R\in[1,\infty]$, the MR incoherence condition (6) is sufficient for the population MR to recover the support uniformly over $\Gamma_{S,\rho,R}$ with slack $\delta$ in the sense of (4). 2. (b)

When $R\in[2,\infty]$ or $\Sigma_{SS}=I$, condition (6) is also necessary.

In the special case where $\Sigma_{SS}=I$, one has the following simpler form of (5):

Lemma 1**.**

Assuming that $\Sigma_{SS}=I$, we have $\Sigma\in\operatorname{MRI}_{S}(\delta^{\prime};R)$ if and only if

[TABLE]

In light of Theorem 1 and Lemma 1, when on-support variables are iid and the non-zero coefficients have zero spread ($R=1$), the Lasso and marginal regression have identical guarantees for support recovery.

The next lemma shows that for unbounded $R$, uniform recovery is not possible by MR except in the trivial case $\Sigma=I_{p}$. Note that for $R=\infty$ according to Theorem 1, MRI is both necessary and sufficient for recovery.

Lemma 2** ($R=\infty$).**

$\operatorname{MRI}_{S}(\delta^{\prime};\infty)=\emptyset$*  for $\delta^{\prime}\geq 1$ and $=\{I_{p}\}$ for $\delta^{\prime}\in[0,1)$.*

The next proposition shows that if the on-support covariance matrix is close enough to singularity, then MRI fails to hold. In other words, MR is sensitive to the smallest eigenvalue of $\Sigma_{SS}$.

Proposition 1**.**

Let $\lambda_{s}^{2}$ be the smallest eigenvalue of $\Sigma_{SS}$, where $\lambda_{s}>0$. Then there exists some $j\in S$ such that for all $k\in S^{c}$,

[TABLE]

As a result, if $\lambda_{s}\leq\delta^{\prime}/(\sqrt{s}(R+1))$ then $\Sigma\not\in\operatorname{MRI}_{S}(\delta^{\prime};R)$ for any $\delta^{\prime}\geq 0$ and $R\geq 1$.

The proofs of Lemma 1 and 2, and Proposition 1 appear in Appendix B. We now consider some examples in which we compare the performance of MR, as controlled by the incoherence introduced in Definition 1 to that of Lasso. For the incoherence parameter controlling the performance of the Lasso, see Definition 3.

We now provide an example which illustrates that MRI could be much more relaxed than PWI, and in the extreme case even match the Lasso incoherence. We assume that the reader is familiar with these two conditions; for details, see (11) and (26) in the appendices. In particular, Lasso incoherence condition, i.e., $|\!|\!|\Sigma_{S^{c}S}\Sigma_{SS}^{-1}|\!|\!|_{\infty}\leq 1-\delta<1$, is a necessary and sufficient condition for (signed) support recovery by the Lasso. In Appendix A, we provide a self-contained proof of this fact.

Example 1** (Pairwise incoherence vs MRI).**

Consider the case where the on-support (i.e., relevant) variables are correlated in groups of size $r$ and the off-support variables are each correlated with at most $r$ of the relevant variables. We compare the bounds that Lasso incoherence (11), MRI, and PWI impose on the tolerable levels of correlation. We show that the Lasso incoherence, as well as MRI, impose an $O(1/r)$ bound on the cross correlations, MRI additionally imposes an $O(1/r)$ bound on the on-support correlations, and PWI imposes an $O(1/s)$ bound on both the on-support and the cross correlations. When $r=2$, i.e. the case of pairwise correlations, and $R=1$, the Lasso and marginal regression reach identical incoherence conditions, while PWI remains quite restrictive.

More precisely, assume that $\Sigma_{SS}$ is block-diagonal with $b$ blocks $(1-\mu)I_{r_{i}}+\mu 1_{r_{i}}1_{r_{i}}^{T},\,i\in[b]$ where $1\leq r:=\max_{i}r_{i}<s$ and $\mu\in[-1/(r-1),1]$. Furthermore111To simplify the calculations, we also assume that for at least one $k$, the support of $\Sigma_{Sk}$ is aligned with one of the blocks in $\Sigma_{SS}$ with size $r$, assume that the support of $\Sigma_{Sk},\,k\in S^{c}$ is equal to $\eta 1_{r}$. Here $1_{r}\in\mathbb{R}^{r}$ is the vector of all ones. The lower bound on $\mu$ is to make $\Sigma_{SS}\succeq 0$. In this case, $\Sigma_{SS}^{-1}$ is block-diagonal with blocks \frac{1}{1-\mu}\big{[}I_{r_{i}}-\frac{\mu}{1-\mu+\mu r_{i}}1_{r_{i}}1_{r_{i}}^{T}\big{]},\,i\in[b]. Then Lasso incoherence (11), with slack $\delta=0$, is equivalent to

[TABLE]

which holds when $|\eta|\leq(1-\mu)/r$, given $\mu\in(-1/(r-1),1)$. Letting $\gamma:=R/(R+1)$, the MR incoherence, with slack $\delta^{\prime}=0$, is a subset of $\gamma(1\pm\eta+(r-1-k)|\mu\pm\eta|+k|\eta|)<1\pm\eta$ for all $0\leq k\leq r-1$. Thus, the MRI condition is

[TABLE]

An interesting case is when $R=1$. We illustrate the conditions for Lasso incoherence, MRI, and PWI in Figure 1.

The observation that even when $\Sigma_{SS^{c}}=0$, MR could fail might seem surprising, but it is a well-known fact related to the idea of faithfulness in graphical models [Spirtes et al., 2000]. For specific choices of $\beta$, one could have $\operatorname{cov}(Y,X_{j})=0$ even when $\beta_{j}\neq 0$ due to the confounding effect of the other variables on the support: $X_{i},i\in S\setminus\{j\}$. This type of “cancellation of correlations” due to confounding factors has been well-documented in the literature. See for example Robins et al. [2003] and Wasserman and Roeder [2009]. Our results, as in Example 1, make precise exactly how much confounding from on-support variables can be tolerated by MR before it fails.

2.2 Marginal regression under sampling

In this section, we analyze the performance of MR on a sample of size $n$ from model (1). In particular, we assume $x_{i}=(x_{i1},\dots,x_{ip})\sim X$ and $y_{i}\sim Y$ i.i.d. for $i=1,\dots,n$ where $(X,Y)$ is distributed as in (1). Note that $x_{i}\in\mathbb{R}^{p}$ and $y_{i}\in\mathbb{R}$. In addition, we assume that the feature and noise vectors are independent Gaussians: $X\sim N(0,\Sigma)$ and $\varepsilon\sim N(0,\sigma^{2}I)$. The sample version of MR replaces the population covariance $r=\operatorname{cov}(X,Y)$ with the sample version:

[TABLE]

For any set $\Gamma\subset\mathbb{R}^{p}$ and $t>0$, let us define

[TABLE]

Lemma 3**.**

Consider model (1) with $\beta\in\Gamma_{S}$ and $\|\beta\|_{2}\leq t$, and assume that $\log p/n\leq C$ for sufficiently small $C>0$. Then, with probability at least $1-2p^{-c_{1}}$,

[TABLE]

Sample MR succeeds in support recovery whenever $\min_{j\in S}|\widehat{r}_{j}|>\max_{j\in S^{c}}|\widehat{r}_{j}|$. Combining Lemma 3 and Theorem 1, we have the following guarantee on the support recovery performance of the sample MR:

Theorem 2** (Sample MR consistency).**

Under the linear model (1) with $\beta\in\Gamma_{S,\rho,R}$ and $\|\beta\|_{2}\leq t$, for any $\rho>0$ and $R\geq 1$, the sample MR recovers the support with probability at least $1-2p^{-c_{1}}$ if

[TABLE]

Although there could be tighter bounds on $\xi(\Sigma;\Gamma_{S},t)$, esp. when $R=1$, here we consider the general bound $\xi(\Sigma;\Gamma_{S},t)\leq|\!|\!|\Sigma_{SS}|\!|\!|_{\text{2}}^{1/2}\min\{t,\sqrt{s}\rho R\}$ which is obtained by noting that

[TABLE]

and that $\|\beta\|_{2}\leq\min\{t,\sqrt{s}\rho R\}$ for any $\beta\in\Gamma_{S}$ with $\|\beta\|_{2}\leq t$. Rewording Theorem 2, using this bound, we have that MR succeeds in support recovery for $\beta\in\Gamma_{S}$ with $\|\beta\|_{2}\leq t$, with probability at least $1-2p^{-c_{1}}$ if

[TABLE]

To observe the typical sample complexity required by (9), assume that $\delta=\Omega(1)$, $|\!|\!|\Sigma_{SS}|\!|\!|_{\text{2}}=O(1)$, $\sigma^{2}=O(1)$ and either of the following two typical scalings of the parameters hold: (a) $\|\beta\|_{2}\asymp 1$ and $|\beta|_{\min}\asymp 1/\sqrt{s}$ hence $t=O(1)$ and $\rho\asymp 1/\sqrt{s}$ or (b) $\|\beta\|_{\infty}\asymp|\beta|_{\min}\asymp 1$, that is, $\rho\asymp R\asymp 1$. In either of these cases, (9) predicts that $n\gtrsim s\log p$ is sufficient for recovery by MR. This is the well-known minimax scaling of the support recovery problem; see Wainwright [2009a].

3 Taxonomy of support recovery conditions

In this section, we offer a broad perspective on the truthfulness conditions governing the performance of several filter and wrapper methods for feature selection. Moreover, we introduce a duality between truthfulness and incoherence conditions for various methods. Our taxonomy further elucidates the strengths and limitations of MR; relates MR to popular feature selectors including Lasso, (Orthogonal) Matching Pursuit, and SIS; and introduces new conjectures and open questions.

The connection between a wrapper method and the filter it uses in scoring the covariates at each iteration extends naturally to their corresponding truthful conditions. Here, we consider MR and the related wrapper methods. For the MR, we are interested in the uniform exact recovery in $\Gamma_{S}$, defined in (2). Noting that under model (1) $\operatorname{cov}(Y,X_{j})=\langle\Sigma_{*j},\beta\rangle$ for any $j$, the truthfulness condition for MR becomes

1. (F1)

$\max_{k\in S^{c}}|\langle\Sigma_{*k},\beta\rangle|<\min_{j\in S}|\langle\Sigma_{*j},\beta\rangle|,\quad\forall\beta\in\Gamma_{S,\rho,R}$.

We call this max-min-$R$ condition as it states that the maximum correlation with an off-support covariate must not exceed the minimum correlation with an on-support one. By definition, (F1) expresses the requirement that MR fully recovers the support. In Theorem 1, we proved that MR incoherence is a necessary and sufficient condition for (F1). Let us separate the case where $R=1$,

1. (F2)

$\max_{k\in S^{c}}|\langle\Sigma_{*k},\beta\rangle|<\min_{j\in S}|\langle\Sigma_{*j},\beta\rangle|,\quad\forall\beta\in\Gamma_{S,\rho,1}$.

This condition, max-min-1, is evidently a weaker condition as it requires uniform recovery over a smaller class of parameters.

For the MP and OMP, following previous literature, one wants conditions for uniform recovery inside $\Gamma_{S,\rho,\infty}$, i.e. all non-zero regression coefficients. The corresponding truthfulness condition is

1. (F3)

$\max_{k\in S^{c}}|\langle\Sigma_{*k},\beta\rangle|<\max_{j\in S}|\langle\Sigma_{*j},\beta\rangle|,\quad\forall\beta\in\mathbb{R}^{p}\backslash\{0\},\beta_{S^{c}}=0$.

This condition, which we call max-max-$\infty$, is clearly necessary and sufficient for the first iteration of the MP and OMP to succeed, i.e., for the first step to select a covariate which is truly on-support. The same condition also guarantees correct selection in the remaining iterations; this can be reasoned by considering the correlation between the residual and the remaining covariates and noting that the required condition for selecting a correct covariate is the same as the inequality in ((F3)) with a different $\beta_{S}$ from $\mathbb{R}^{s}\backslash\{0\}$ (ct. Remark 1 in Appendix B.4). In Tropp [2004] the author shows one side of the following equivalence:

Lemma 4**.**

(F3)* is equivalent to the Lasso incoherence condition: $|\!|\!|\Sigma_{S^{c}S}\Sigma_{SS}^{-1}|\!|\!|_{\infty}<1$.*

The other side follows easily and, for completeness, we provide a simple proof of both sides in Appendix B. A consequence of this equivalence is that (F3) can serve as a truthfulness condition for the Lasso. Another way to see this is to consider forward-stagewise regression, which is known to be a fast implementation of the Lasso. In this implementation, the covariates with the highest correlation are exploited to update the residual with a fixed step size. The necessity and sufficiency of  (F3) for the Lasso thus can be argued by noting the similarity between the MP and forward-stagewise regression.

It is not clear how (F3) connects to (F1) or (F2). In particular, we do not know if one requires more restrictions on the covariance matrix than the others. However, based on our simulations, we conjecture that (F2) implies (F3), i.e. if for a particular covariance matrix, the marginal regression can recover the support for all $\beta_{S}\in\{+1,-1\}^{s}$, then the Lasso is guaranteed to recover the support for all non-zero $\beta$ with support $S$. If this relation holds then the marginal regression could be no stronger than the Lasso in uniform sparse recovery, even under conditions that favor MR ($R=1$).

Conjecture 1**.**

(F2)* implies (F3).*

To give partial evidence of this conjecture, we show that max-min-$R$ when $R=6$ for all subsets $S$ of size $s$ implies Restricted Isometry Property (RIP), a condition known to be sufficient for the close relative of the Lasso known as the basis pursuit regression; see (25) in Appendix B.5. The RIP is defined as follows: (cf. Wainwright [2018])

Definition 2**.**

For $s\in[p]$, the covariance matrix $\Sigma$ satisfies a restricted isometry property (RIP) of order $s$ with constant $\delta_{s}(\Sigma)>0$ if for all subsets $S$ of size at most $s$, one has $|\!|\!|\Sigma_{SS}-I_{s}|\!|\!|_{\text{2}}\leq\delta_{s}(\Sigma).$

A sufficient condition for the exact parameter recovery by the basis pursuit is an RIP condition on $\Sigma$ of order $2s$, namely, $\delta_{2s}(\Sigma)\leq 1/3$. Let us define $\Gamma_{(s)}:=\bigcup_{S:\;|S|\,\leq\,s}\Gamma_{S}$ where $\Gamma_{S}$ is given in (2). An incoherence condition holds over $\Gamma_{(s)}$ iff it holds over $\Gamma_{S}$ for all subset $S$ of size at most $s$. The following propositions relates RIP to the MR incoherence derived in Section 2.1. The proofs are deferred to Appendix B.5.

Proposition 2**.**

Assume $\operatorname{diag}(\Sigma)=1_{p}$. If MR incoherence (5) holds over $\Gamma_{(s)}$, then

[TABLE]

This proposition roughly shows that in terms of the strength, the MR incoherence is stronger than RIP, which is in turn comparable in strength to the Lasso incoherence.

Finally, we introduce the truthfulness condition relevant to SIS, which progressively discards irrelevant covariates (conservatively) with the aim of keeping all the relevant ones among the remaining covariates at all times. To achieve this goal, assuming that we want to discard at most $d$ covariates and do so by removing one covariate at a time, one requires that the $d$ covariates with least correlations with the target are “off” the support, so that we are guaranteed not to discard any relevant covariate at any stage. Thus, the necessary condition for SIS to achieve uniform recovery over all $\beta\in\Gamma_{S,\rho,R}$ is:

1. (F4)

$\min_{k\in S^{c}}|\langle\Sigma_{*k},\beta\rangle|<\min_{j\in S}|\langle\Sigma_{*j},\beta\rangle|,\quad\forall\beta\in\Gamma_{S,\rho,R}$.

Formalizing an incoherence condition for (F4) similar to what we have for the other three cases would be of great interest. Note that (F4) is the least stringent of the four conditions we introduced. In other words, we expect (F4) to impose the least constraints on the covariance matrix. As far as we know, little is known about the exact nature of the conditions (F4) imposes.

Figure 2 summarizes the discussion in this section. It illustrates the relation among the truthfulness conditions and their corresponding incoherence conditions for the MR, (O)MP, Lasso, and the SIS. This taxonomy of truthfulness and incoherence conditions can provide a platform to compare other subclasses of filter and wrapper methods. We hope it can benefit the future research in sparse recovery.

4 Discussion and extensions

We studied support recovery performance of marginal regression (MR) and obtained a near complete characterization of the conditions for recovery in terms of the covariance matrix of the features. We introduced parameter $R$ measuring the spread of the coefficients and showed that when $R\geq 2$ or $\Sigma_{SS}=I$, the MR incoherence conditions (Definition 1) are necessary and sufficient. We have an example (not mentioned in the paper for brevity) that these conditions are not necessary otherwise (but still sufficient). An open question is whether the truthfulness of MR implies Lasso incoherence, which is what we conjecture. If true, this settles the question of the dominance of Lasso over MR. Overall, our theory provides a more optimistic view of MR for feature selection and provides some long overdue insights into its strengths and limitations. We have also provided a framework to study the truthfulness conditions for general filter and iterative wrapper methods.

Finally, we have laid the foundation to extend the MR incoherence condition to non-linear filter methods. Indeed, the core results we developed here about the performance of MR can be readily extended to a more general framework. We sketch the steps towards a nonlinear extension here and leave the rigor to a later work. The plan is to show that MR or any other filter that has a semi-norm property w.r.t. to its arguments (such as tree-based impurity reduction scores when training decision trees) can benefit a performance guarantee similar in nature to that of Theorem 1, under a general sparse additive model with a sufficiently restricted function class.

Here, we briefly sketch the argument. Consider a general additive model, in which $Y=\sum_{j\in S}f_{j}(X_{j})+\varepsilon$, with similar assumptions about $X,\varepsilon$ as in (1). Suppose that the function tuple $f=(f_{j})_{j\in S}$ is from the following class subclass of $\mathcal{F}_{0}=\{f:\;\;f^{\prime}\in L^{2}(a,\infty)\}$:

[TABLE]

where $Y_{j,k}^{\lambda}:=\mu_{X_{S},X_{j}}+\lambda\mu_{X_{S},X_{k}}$ with $\mu_{X_{S},X_{j}}=(\mu_{X_{i},X_{j}},i\in S)$ and $\mu_{X_{i},X_{j}}(t):=\mathbb{E}[X_{i}1\{t\leq X_{j}\}]$. One can show for functions in this class that $|\operatorname{cov}(X_{j}+\lambda X_{k},f_{i}(X_{i}))|$ is bounded away from zero and also bounded above. Then using the triangle inequality and sub-linearity of $|\operatorname{cov}(\cdot,Z)|$, one can replicate the argument for Theorem 1. The only obstacle is to show the class of function tuples in (10) is non-trivial. We believe this to be true for sufficiently regular classes of functions. Furthermore, we need to characterize the corresponding class for other semi-norm filter methods, allowing us to replace the Pearson correlation operator $|\operatorname{cov}(\cdot,Z)|$ with a general seminorm filter. We also observe that the role played in our arguments by the spread of the coefficients, the $R$ parameter, will be played by the bounds on the spread of the derivatives of the underlying functions.

Appendix A Lasso incoherence

In this section, we recall the Lasso incoherence condition and show that it is indeed necessary and sufficient for the Lasso to perform exact “support plus sign recovery” at the population level. Though this result is more or less known [Wainwright, 2018], there is some nuance to the statement we give here at the population level; in particular, we could not find a result that covers the necessity of Lasso incoherence (even at the population level) as stated here, which is part (a) of Proposition 3. We also provide a short self-contained proof of this result for completeness. We then compare the MR incoherence with some well-known incoherences that have appeared in the literature surrounding Lasso and its close relative, the basis pursuit.

A.1 Lasso at the population level

Let us start by stating the incoherence condition Lasso is sensitive to:

Definition 3**.**

Lasso incoherence (LAI) condition with slack $\delta$, for recovering $S\subset[p]$, is

[TABLE]

and we write $\Sigma\in\operatorname{LAI}_{S}(\delta)$ if this condition holds. We also write $\Sigma\in\operatorname{LAI}_{S}(0+)$ if the condition holds for some $\delta\in(0,1)$.

An alternative way of writing condition (11) is $\max_{k\,\in\,S^{c}}\|\Sigma_{SS}^{-1}\Sigma_{Sk}\|_{1}\leq 1-\delta$. It is well known that (11) controls the model selection performance of the Lasso. Consider the population Lasso

[TABLE]

where $X\in\mathbb{R}^{p}$ and $Y=X^{T}\beta+\varepsilon\in\mathbb{R}$ are as in model (1). We say that $\widetilde{\beta}$ is sign selection consistent if in addition to $\operatorname{supp}(\widetilde{\beta})=\operatorname{supp}(\beta)=:S$, we have $\operatorname{sign}(\widetilde{\beta}_{S})=\operatorname{sign}(\beta_{S})$. Recall the definition of $\Gamma_{S,\rho,R}$ from (2) and note that $\Gamma_{S,0,\infty}$ is the set of all vectors $\beta\in\mathbb{R}^{p}$ whose support is contained in $S$. We say that a subset $\mathcal{B}_{S}\subset\Gamma_{S,0,\infty}$ is sign rich if it contains all the sign patterns over $S$, i.e., every $\beta\in\mathbb{R}^{p}$ with $\beta_{S^{c}}=0$ and $\beta_{S}\in\bigcup_{\rho>0}\{-\rho,\rho\}^{s}$ belongs to $\mathcal{B}_{S}$.

The following proposition shows that $\Sigma\in\operatorname{LAI}_{S}(0)$ is essentially necessary and sufficient for the population Lasso to be sign selection consistent over sign rich sets of parameters.

Proposition 3**.**

Fix some $S\subset[p]$ and assume that $\Sigma_{SS}$ is nonsingular. Let $\mathcal{B}_{S}$ be any sign pattern rich subset of $\Gamma_{S,0,\infty}$. Then, under model (1) for $(Y,X)$ with $\beta$ set to $\beta^{*}$:

• (a)

If for every $\beta^{*}\in\mathcal{B}_{S}$, the population Lasso in (12) with input $(Y,X)$ and some $\lambda>0$ is sign selection consistent, then $\Sigma\in\operatorname{LAI}_{S}(0)$.

• (b)

If $\Sigma\in\operatorname{LAI}_{S}(0+)$, then for every $\beta^{*}\in\mathcal{B}_{S}$, the Lasso with input $(Y,X)$ and sufficiently small $\lambda>0$, has a unique solution which is sign selection consistent.

Proposition 3 is more of less colloquially known and can be traced back to the work of Fuchs [2005], Tropp [2006], Zhao and Yu [2006], Wainwright [2009b], Genovese et al. [2012] among others. It is not often stated in the population form presented here, and we give a proof in Appendix B. Note that part (a) states something fairly strong: As long as the set of $\beta$s over which we require uniform (sign) selection consistency contains all possible sign patterns, then Lasso incoherence is necessary, no matter how small a value of $\lambda>0$ we choose. Sign rich sets, for example, include $\Gamma_{S,\rho,R}$ for any $R\in[1,\infty]$. The interesting aspect of the population level result is that Lasso is still severely restricted (even with infinite sample size that is) in terms of how much correlation among features it can handle. Contrast this with the ordinary least-squares (or ridge regression), where $\beta$ itself can be recovered exactly at the population level, for any nonsingular covariance matrix $\Sigma$.

Appendix B Proofs

B.1 Proof of Theorem 1

Under model (1) $\operatorname{cov}(X,Y)=\mathbb{E}X(X^{T}\beta+\varepsilon)=\Sigma\beta$, that is $\operatorname{cov}(X_{j},Y)=\langle\Sigma_{*j},\beta\rangle$ where $\Sigma_{*j}$ is the $j$th column of $\Sigma$. If follows that MR achieves exact support recovery over $\Gamma_{S}$ with slack $\delta$ in the sense of (4) if and only if

[TABLE]

Lemma 5**.**

For any $a,b\in\mathbb{R}$ and $\delta>0$, we have $|a|-|b|>\delta\iff|a-\lambda b|>\delta,\;\forall\lambda\in[-1,1].$

Using Lemma 5, condition (13) is equivalent to $|\langle\Sigma_{*j}+\lambda\Sigma_{*k},\beta\rangle|>\delta$ for all $\beta\in\Gamma_{S},\;j\in S,\;k\in S^{c}$ and $\lambda\in[-1,1]$. For a set $\Gamma\subset\mathbb{R}^{p}$, we define its absolute dual with slack $\delta$ to be

[TABLE]

Thus, exact support recovery by MR over $\Gamma_{S}$ with slack $\delta$ is equivalent to

[TABLE]

The following technical lemma, proved in Appendix B, characterizes the absolute dual of the parameters space $\Gamma_{S}$:

Lemma 6**.**

The absolute dual of $\Gamma_{S}$ given in (2) can be written as $\Gamma_{S}^{\dagger}=\Gamma_{S}^{\prime}\cup\Omega_{S}$ where

[TABLE]

Theorem 1(a).

Let $\delta^{\prime}=\delta/\rho$. By Lemma 6, $\Gamma^{\prime}_{S}\subset\Gamma_{S}^{\dagger}$. Hence, replacing $\Gamma_{S}^{\dagger}$ in (15) with $\Gamma_{S}^{\prime}$ we get the following sufficient condition for recovery:

[TABLE]

We have (see Appendix B for the proof):

Lemma 7**.**

Condition (16) implies $\|\Sigma_{Sj}+\lambda\Sigma_{Sk}\|_{\infty}=\Sigma_{jj}+\lambda\Sigma_{jk}$ for all $\lambda\in[-1,1]$.

Therefore, (16) implies

[TABLE]

In the other direction, (17) clearly implies (16) noting that $\|\Sigma_{Sj}+\lambda\Sigma_{Sk}\|_{\infty}\geq$ $\Sigma_{jj}+\lambda\Sigma_{jk}\geq 0$ for all $|\lambda|\leq 1$, since by assumption $\Sigma_{jj}=1$ hence $|\Sigma_{jk}|\leq 1$. Since the RHS of (17) is convex in $\lambda$ and the LHS is linear, (17) holds if and only if it holds at the two endpoints $\lambda=-1,1$ which gives (5) as desired. ∎

Theorem 1(b), case $R\geq 2$.

Assume $\rho=1$ without loss of generality. We also write $\Gamma_{S,R}=\Gamma_{S}$ to emphasize the dependence on $R$. For any $\phi\in\mathbb{R}^{p}$, let us define

[TABLE]

By definition, $\phi$ belongs to $\Gamma_{S,R}^{\dagger}$ if and only if $\alpha_{R}(\phi)>\delta$. An interesting case is when $R=1$, in which case we use the notation $\alpha(\phi)$ instead of $\alpha_{1}(\phi)$. The minimization in the case of $R=1$ is over $\Gamma_{S,1}$, which consists of vectors with $\pm\rho$ coordinates on the support. For any $\beta\in\Gamma_{S,1}$, it is easy to see that

[TABLE]

where $S_{1}=\{j\in S\;:\;\operatorname{sign}(\beta_{j})=\operatorname{sign}(\phi_{j})\}$ and $S_{0}=S\backslash S_{1}$. The relation works both ways, i.e., for any partition of $S$ into $S_{1}$ and $S_{0}$, there exists a $\beta\in\Gamma_{S,1}$ that (18) holds.

Let $\{S_{0}(\phi),S_{1}(\phi)\}$ be any partition that achieves the minimum for $\phi$ such that $S_{1}(\phi)$ corresponds to the larger of the two sums, that is

[TABLE]

Note that whenever $\phi\neq 0$, we have $|S_{1}(\phi)|\geq 1$ with $|\phi_{j}|>0$ for at least some $j\in S_{1}(\phi)$. We adopt the convention of putting the indices of the zero coordinates of $\phi$ in $S_{0}(\phi)$. Thus, for $\phi\neq 0$, we have $\min_{j\in S_{1}(\phi)}|\phi_{j}|>0$. As the following lemma states, $\alpha(\phi)$ can be used to approximate $\alpha_{R}(\phi)$.

Lemma 8**.**

If $\phi\in\Gamma_{S,R}^{\dagger}\setminus\Gamma_{S,R}^{\prime}$, then  \alpha_{R}(\phi)\leq\max\big{\{}0,(2-R)\,\alpha(\phi)\big{\}}.

As a consequence, $\Gamma_{S}^{\dagger}=\Gamma_{S}^{\prime}$ whenever $R\geq 2$, completing the proof of part (b). ∎

Theorem 1(b), case $\Sigma_{SS}=I$.

Fix $k\in S^{c}$. Let $j^{*}\in\text{argmin}_{j^{\prime}\in S}|\Sigma_{j^{\prime}k}|$ and choose $\beta\in\Gamma_{S,\rho,R}$ such that

[TABLE]

Selection consistency implies that

[TABLE]

using assumption $\Sigma_{SS}=I$ in the last equality. Dividing by $\rho$ and rearraging proves (7) which is equivalent to (6), due to Lemma 1. ∎

B.2 Proofs of auxiliary lemmas for Theorem 1

Lemma 7.

Fix $j\in S,\;k\in S^{c}$ and let $u(\lambda)=\Sigma_{Sj}+\lambda\Sigma_{Sk}\in\mathbb{R}^{s}$. Define

[TABLE]

where $u_{j}(\lambda)=\Sigma_{jj}+\lambda\Sigma_{jk}$ is the $j$th component of $u(\lambda)$. The set $\Omega$ contains the roots of the function $\lambda\mapsto\|u(\lambda)\|_{\infty}-u_{j}(\lambda)$, and since this function is continuous, $\Omega$ is a closed set. If we also prove that $\Omega$ is an open set in $[-1,1]$, then it must be either empty or the whole interval. It is not empty since it contains [math] (recall the assumption $\Sigma_{jj}=1$). In the following, we show that $\Omega$ is open in $[-1,1]$. Let

[TABLE]

Clearly, $E$ is open in $[-1,1]$. We plan to show that $E=\Omega$. First, we have $E\subset\Omega$ since

[TABLE]

implying that $u_{j}(\lambda)=\|u(\lambda)\|_{\infty}$, that is $\lambda\in\Omega$. Assuming (16), we also have $\Omega\subset E$. We conclude that $E=\Omega$, hence $\Omega$ is open and should contain all of $[-1,1]$. As a result, (16) implies (17) and the proof is complete. ∎

Lemma 8.

Fix $\phi\in\Gamma_{S,R}^{\dagger}\setminus\Gamma_{S,R}^{\prime}$ and let $S_{1}=S_{1}(\phi)$ and $S_{0}=S_{0}(\phi)$ be as defined in (19). According to Lemma 6, $\phi\in\Gamma_{S}^{\prime\prime}$ (and $\phi\neq 0$). This implies that $|S_{1}|>1$. Otherwise, $|S_{1}|=1$, and $S_{1}$ should consist of the index of a maximal element of $\phi$ in absolute value, hence

[TABLE]

a contradiction. The last inequality is a consequence of $\phi\in\Gamma_{S}^{\prime\prime}$.

Now, let $c=\sum_{j\in S_{0}}|\phi_{j}|$. Then, $\alpha(\phi)=\sum_{j\in S_{1}}|\phi_{j}|-c$. Let $j^{*}=\text{argmin}_{j^{\prime}\in S_{1}}|\phi_{j^{\prime}}|$ and note that $|\phi_{j^{*}}|>0$ by construction. Then,

[TABLE]

where we have used $|S_{1}|>1$. Now, let $S_{1}^{\prime}=S_{1}\backslash\{j^{*}\},\;S_{0}^{\prime}=S_{1}\cup\{j^{*}\}$. We have

[TABLE]

where the inequality is by the optimality of $\{S_{0},S_{1}\}$ partition. Since $|\phi_{j*}|>0$, this implies $\alpha(\phi)\leq|\phi_{j^{*}}|$. Combining with (20), we have $\alpha(\phi)\leq c$.

For any $1\leq\gamma\leq R$, define $\beta^{(\gamma)}\in\Gamma_{S,R}$ such that \beta^{(\gamma)}_{j}=\operatorname{sign}(\phi_{j})\big{[}1\{j\in S_{1}\}-\gamma\;1\{j\in S_{0}\}\big{]}, for all $j\in S$. Then,

[TABLE]

where the equality follows since $\langle\phi,\beta^{(\gamma)}\rangle$ is a decreasing and continuous function of $\gamma$. But

[TABLE]

where we have used $\alpha(\phi)\leq c$. The proof is complete. ∎

B.3 Proof of Theorem 2

Let us write $\operatorname{MRI}_{S}(\delta^{\prime})=\operatorname{MRI}_{S}(\delta^{\prime};R)$ for simplicity. The condition for the sample MR to recover the support is $|\widehat{r}_{j}|>|\widehat{r}_{k}|$ for all $j\in S$ and $k\in S^{c}$. By triangle inequality, $|\widehat{r}_{j}|\geq|r_{j}|-\|\widehat{r}-r\|_{\infty}$ and $|\widehat{r}_{k}|\leq|r_{k}|+\|\widehat{r}-r\|_{\infty}$. Thus sample MR is consistent if $|r_{j}|>2\|\widehat{r}-r\|_{\infty}+|r_{k}|$ for all $j\in S$ and $k\in S^{c}$ which is equivalent to population MR consistency with slack $\delta:=2\|\widehat{r}-r\|_{\infty}$. Theorem 1 thus gives the following sufficient condition

[TABLE]

where the second inequality holds with probability $\geq 1-2p^{-c_{1}}$ by Lemma 3 and the fact that $\operatorname{MRI}_{S}(\delta^{\prime}_{1})\subset\operatorname{MRI}_{S}(\delta^{\prime}_{2})$ whenever $\delta^{\prime}_{1}\leq\delta^{\prime}_{2}$.

B.4 Other proofs

Lemma 1.

Fix $j\in S,\;k\in S^{c}$. Assumption (7) implies $R\|\Sigma_{Sk}\|_{1}<1-\delta^{\prime}+(R-1)|\Sigma_{jk}|$ or equivalently

[TABLE]

Replacing $-|\Sigma_{jk}|$ with $\pm\Sigma_{jk}$, adding $R(1\pm\Sigma_{jk})$ to both sides, and using $\Sigma_{j^{\prime}j}=1\{j^{\prime}=j\}$,

[TABLE]

which gives the desired result after some algebra. The only if part holds as the previous argument is reversible. ∎

Lemma 2.

Fix $\delta^{\prime}\geq 0$. Assume $\Sigma\in\operatorname{MRI}_{S}(\delta^{\prime};R)$, for all $R<\infty$, or equivalently

[TABLE]

The feasibility of these conditions for every $R\in[1,\infty)$ requires that

[TABLE]

which implies that $\Sigma_{j^{\prime}k}=\Sigma_{j^{\prime}j}=0$ for all $j,j^{\prime}\in S,\;k\in S^{c}$ that $j\not=j^{\prime}$. Therefore, $\Sigma_{S^{c}S}=0$ and $\Sigma_{SS}=I$. Moreover, $\delta^{\prime}$ must be less than one. This proves the desired result. ∎

Proposition 1.

Let $\mu$ be a unit norm eigenvector of $\Sigma_{SS}$ associated with eigenvalue $\lambda_{s}^{2}$. Then, $\operatorname{var}(\mu^{T}X_{S})=\mu^{T}\Sigma_{SS}\,\mu=\lambda_{s}^{2}$ and as a result,

[TABLE]

Pick $j=\text{argmax}_{j^{\prime}\in S}|\mu_{j^{\prime}}|$ and let $S^{\prime}:=S\setminus\{j\}$. For any $k\in S^{c}$,

[TABLE]

where $(i)$ is by the triangle inequality $|a\pm c|-|b\pm d|\leq|a+b|+|c+d|$. Rearranging and noting that $|\mu_{j}|>0$ (since $\mu\neq 0$), we have with $\nu_{S^{\prime}}:=\mu_{S^{\prime}}/|\mu_{j}|$,

[TABLE]

since $\|v_{S^{\prime}}\|_{\infty}\leq 1$ which holds by the particular choice of $j$. Adding $|\Sigma_{jj}\pm\Sigma_{jk}|$ and dividing by 2,

[TABLE]

Since $\|\mu\|_{2}=1$, we have $\|\mu\|_{\infty}\geq 1/\sqrt{s}$ which gives the desired inequality. The last assertion of the proposition follows form the inequality by noting that $R/(R+1)\geq 1/2$. ∎

Lemma 3.

We have $Y\sim N(0,\beta^{T}\Sigma\beta+\sigma^{2})$ and $X_{j}\sim N(0,1)$. It follows that $\|Y\|_{\psi_{2}}\lesssim\sqrt{\beta^{T}\Sigma\beta+\sigma^{2}}\leq\sqrt{\beta^{T}\Sigma\beta}+\sigma$ and $\|X_{j}\|_{\psi_{2}}\lesssim 1$. Then, by [Vershynin, 2016, Lemma 2.7.7] $YX_{j}$ is sub-exponential and

[TABLE]

using $\beta\in\Gamma_{S}$ and $\|\beta\|_{2}\leq t$ by assumption and definition (8). By centering [Vershynin, 2016, Exercise 2.7.10], we have the same bound, up to constants, for $\|YX_{j}-\mathbb{E}[YX_{j}]\|_{\psi_{1}}\lesssim\|YX_{j}\|_{\psi_{1}}\lesssim\xi^{\prime}$. We note that $\widehat{r}_{j}-r_{j}=\frac{1}{n}\sum_{i=1}^{n}y_{i}x_{ij}-\mathbb{E}[y_{i}x_{ij}]$ which is an average of iid centered sub-exponential variables distributed as $YX_{j}-\mathbb{E}[YX_{j}]$. Bernstein inequality for sub-exponential variables [Vershynin, 2016, Theorem 2.8.1] implies that for any fixed $j\in[p]$,

[TABLE]

Applying union bound over $j=1,\dots,p$,

[TABLE]

Taking $\tau=\sqrt{(1+c_{1})\log p/(cn)}\leq 1$, where the inequality holds by the assumption that $\log p/n$ is sufficiently small, we obtain the result. ∎

Lemma 5.

The result follows from the following identity

[TABLE]

where $x_{+}:=\max(x,0)=x1\{x>0\}$ is the positive part of $x$. To see identity (21) note that it holds trivially for $b=0$. Now, assume $b\neq 0$, and let $\alpha=a/b$. Then,

[TABLE]

Multiplying by $|b|$ gives (21). ∎

Lemma 6.

First, we prove $\Gamma_{S}^{\prime}\subseteq\Gamma_{S}^{\dagger}$. Take any $\beta\in\Gamma_{S}$ and $\phi\in\Gamma_{S}^{\prime}$. Let $j\in\text{argmax}_{j^{\prime}\in S}\;|\phi_{j^{\prime}}|$. Then,

[TABLE]

where (i) and (iii) are implied by $\beta\in\Gamma_{S}$ and (ii) by $\phi\in\Gamma_{S}^{\prime}$. Notice that the second to last inequality is strict.

In order to prove the second part of lemma, fix some $\phi\not\in\Gamma_{S}^{\prime}\cup\Gamma_{S}^{\prime\prime}$. We show that $\phi\not\in\Gamma_{S}^{\dagger}$, by constructing some $\beta\in\Gamma_{S}$ such that $|\langle\phi,\beta\rangle|\leq\delta$. Let $\delta^{\prime}=\delta/\rho$. With some algebra, we have

[TABLE]

Let us define the following mutually exclusive intervals:

[TABLE]

We consider four cases corresponding to $\|\phi_{S}\|_{1}-\|\phi_{S}\|_{\infty}\in I_{i}$, for $i=1,2,3,4$. We construct $\beta^{1},\beta^{2},\beta^{3},\beta^{4}\in\mathbb{R}^{p}$ such that whenever $\|\phi_{S}\|_{1}-\|\phi_{S}\|_{\infty}\in I_{i}$ then $\beta^{i}\in\Gamma_{S}$ and $|\langle\phi,\beta^{i}\rangle|\leq\delta^{\prime}$. Let us proceed with the construction. We set $\beta^{i}_{S^{c}}=\bar{0}_{p-s}$ for all $i=1,2,3,4$ and let $j\in\text{argmax}_{j^{\prime}\in S}\;|\phi_{j^{\prime}}|$. Define $\beta^{i}_{j}$ as follows:

[TABLE]

and for $j^{\prime}\in S\backslash\{j\}$, let

[TABLE]

We leave it to the reader to verify that for $i=1,4$ we have $0\leq\langle\phi,\beta^{i}\rangle\leq\delta^{\prime}$ and for $i=2,3$, we have $\langle\phi,\beta^{i}\rangle=0$. Furthermore, one should check that $\beta^{2},\beta^{3}\in\Gamma_{S}$.∎

Remark 1**.**

The proof is by induction. Suppose the selected covariates after some iterations are $X_{j},\,j\in\widehat{S}$ and $\widehat{S}\subset S$. Also assume the residual is

[TABLE]

We don’t make any assumption about the origin of $\gamma_{j}$ so the rest of proof works for MP, OMP, and forward-stagewise regression. Denote $\gamma=(\gamma_{j})_{j\in\widehat{S}}$. Now let us compute the covariance of $R$ and each covariate, $X_{j},\,j\in[p]$:

[TABLE]

where $\bar{\gamma}\in\mathbb{R}^{p}$ satisfies $\bar{\gamma}_{\widehat{S}}=\gamma$ and $\bar{\gamma}_{\widehat{S}^{c}}=0$. Since $\beta-\bar{\gamma}$ has support $S$ and it is non-zero (unless the residual is zero and we stop the iteration), the condition (F3) guarantees the next selection would be also from the support.

Lemma 4.

If $\Sigma_{SS}$ is singular, neither of (F3) or Lasso incoherence holds. Therefore assume $\Sigma_{SS}$ is non-singular,

[TABLE]

The first equivalence holds since the linear operator defined by $\Sigma_{SS}$ is a bijection on $\mathbb{R}^{s}\backslash\{0\}$ and the second equivalence holds by definition of the operator norm $|\!|\!||\!|\!|_{\infty}$. ∎

Proposition 3.

Let us write $\Sigma_{xy}=\mathbb{E}[XY]$ and $\Sigma_{xx}=\Sigma=\mathbb{E}[XX^{T}]$. The Lasso solution can be written as

[TABLE]

The optimality conditions are obtained by requiring that zero belongs to the subdifferential of the objective, i.e., $\Sigma_{xx}\widetilde{\beta}-\Sigma_{xy}+\lambda u$ where $u\in\partial\|\widetilde{\beta}\|_{1}$. Alternatively, $u=\operatorname{sign}(\widetilde{\beta})$ where $\operatorname{sign}$ should be interpreted as a generalized sign vector (i.e., for the scalar version $\operatorname{sign}(x)\in[-1,1]$ whenever $x=0$). Under model (1) with $\beta=\beta^{*}$, we have $\Sigma_{xy}=\Sigma\beta^{*}$. The optimality conditions are given by

[TABLE]

Let us write $\Delta=\widetilde{\beta}-\beta^{*}$. Consider part (a) first: Assume that $\widetilde{\beta}$ has the correct support, so that $\widetilde{\beta}_{S^{c}}=0$, and since $\beta^{*}_{S^{c}}$, we have $\Delta_{S^{c}}=0$. Partitioning over $S$ and $S^{c}$, we have

[TABLE]

that is, $\Delta_{S}=-\lambda\Sigma_{SS}^{-1}u_{S}$ and $\Sigma_{S^{c}S}\Delta_{S}+\lambda u_{S^{c}}=0$. Substituting the expression for $\Delta_{S}$ from the first equation into the second, we obtain (using $\lambda>0$)

[TABLE]

Since $u_{S^{c}}$ is a generalized sign vector, we have $\|u_{S^{c}}\|_{\infty}\leq 1$. Assuming that $\widetilde{\beta}_{S}$ has the correct sign, we get $u_{S}=\operatorname{sign}(\widetilde{\beta}_{S})=\operatorname{sign}(\beta^{*}_{S})$. As $\beta^{*}_{S}$ varies in $\mathcal{B}_{S}$, $\operatorname{sign}(\beta^{*}_{S})$ takes all possible values in $\{\pm 1\}^{s}$. Thus, under the assumption of part (a), we have

[TABLE]

The LHS is equal to $|\!|\!|\Sigma_{S^{c}S}\Sigma_{SS}^{-1}|\!|\!|_{\infty}$ completing the proof of part (a). (Note that the maximum of a convex function over a set is equal to its maximum over the convex hull of that set, hence $\sup_{u_{S}\,\in\,\{\pm 1\}^{s}}f(u_{S})=\sup_{u_{S}\in\operatorname{\mathbb{B}}_{\infty}}f(u_{S})$ for any convex $f$.)

For part (b), consider a candidate $\widetilde{\beta}$ with $\widetilde{\beta}_{S^{c}}=0$ and $\widetilde{\beta}_{S}=\beta^{*}_{S}-\lambda\Sigma_{SS}^{-1}u_{S}$. Choose $u_{S}$ such that it satisfies

[TABLE]

which always has a solution for sufficiently small $\lambda>0$. In fact, for sufficiently small $\lambda$, we obtain $u_{S}=\operatorname{sign}(\beta^{*}_{S})$. Define $u_{S^{c}}$ as in (24). This is a valid choice since $\|u_{S^{c}}\|_{\infty}\leq|\!|\!|\Sigma_{S^{c}S}\Sigma_{SS}^{-1}|\!|\!|_{\infty}\|u_{S}\|_{\infty}<1$ using the assumption of part (b) and that $\|u_{S}\|_{\infty}\leq 1$. It follows that this dual vector is strictly feasible. The constructed pair $(\widetilde{\beta},u)$ is then feasible and satisfies the optimality condition (23), hence it is an optimal primal-dual pair; in addition strict dual feasibility implies uniqueness of the primal solution. Hence, constructed $\widetilde{\beta}$ is the unique solution of the Lasso, with correct support and correct sign $\operatorname{sign}(\widetilde{\beta}_{S})=u_{S}=\operatorname{sign}(\beta^{*}_{S})$. ∎

B.5 Comparison with other incoherence conditions

Let us recall some other incoherence conditions that are often considered when studying the basis pursuit, a close relative of Lasso, which at population level solves the optimization problem:

[TABLE]

The following incoherence condition is well-known Wainwright [2018]:

Definition 4**.**

Pairwise incoherence (PWI) parameter of the covariance matrix $\Sigma$ is defined as

[TABLE]

It is known that if $\Sigma$ has pairwise incoherence $\delta_{\text{PW}}(\Sigma)\leq 1/(3s)$, then the basis pursuit problem (25) recovers the (true) vector of parameters, $\beta$. Let us define

[TABLE]

An incoherence condition holds over $\Gamma_{(s)}$ iff it holds over $\Gamma_{S}$ for all subset $S$ of size at most $s$.

We have the following result connecting PWI and MR incoherence:

Proposition 4**.**

Assume $\operatorname{diag}(\Sigma)=1_{p}$. Then, MR incoherence (5) holds over $\Gamma_{(s)}$ if

[TABLE]

Proposition 4 together with Proposition 2 stated in Section 3 relate two well-known conditions to the MR incoherence derived in Section 2.1. These two propositions roughly show that in terms of the strength, the MR incoherence is somewhere between the PW incoherence and the RIP, that is, PW incoherence implies a form of MR incoherence which in turn implies a form of RIP.

Proof of Proposition 4.

Taking $j\in S$ and $k\in S^{c}$,

[TABLE]

where the first inequality is by assumption and the second uses $\Sigma_{jj}=1$. The MR incoherence (5) follows by adding \big{|}\Sigma_{jj}\pm\Sigma_{jk}\big{|}=\Sigma_{jj}\pm\Sigma_{jk} to both sides and multiplying by $R/(R+1)$. ∎

Proof of Proposition 2.

Pick some $\widetilde{S}\subset[p]$ with size $2s$. Choose a balanced partition, $S_{0},S_{1}$, of $\widetilde{S}$, i.e. $|S_{0}|=|S_{1}|$. The MR incoherence condition holds for both $S=S_{1}$ and $S=S_{2}$:

[TABLE]

Fix some $i\in\{0,1\}$, $j\in S_{i}$ and $k\in S_{1-i}$. By the convexity of $\ell_{1}$ norm,

[TABLE]

Let $\Delta=\Sigma-I_{p}$. Note that $\Delta_{S_{i}S_{i}}=\Sigma_{S_{i}S_{i}}-I_{s}$, hence

[TABLE]

by (28) and the assumption $\operatorname{diag}(\Sigma_{S_{i}S_{i}})=1_{s}$. Let $e_{j}$ be the $j$th basis vector of $\mathbb{R}^{s}$. Then, $\Delta_{S_{i}j}\pm\Delta_{S_{i}k}=\Sigma_{S_{i}j}\pm\Sigma_{S_{i}k}-e_{j}$, hence

[TABLE]

using $\Sigma_{jj}=1$, (27) and $\Sigma_{jk}=\Delta_{jk}$.

Using a convexity argument as in (28), we obtain $\|\Delta_{S_{i}k}\|_{1}<(1-\delta^{\prime})/R+|\Delta_{jk}|$. Taking the sum over $j\in S_{i}$ and rearranging, we have

[TABLE]

Since (29) and (30) hold for any $j\in S_{i}$ and $k\in S_{1-i}$, we have shown that every column of $\Delta_{S_{i}S_{i}}$ and $\Delta_{S_{i}S_{1-i}}$, for $i=0,1$, has $\ell_{1}$ norm bounded by $(1-\delta^{\prime})/R$ and $s(1-\delta^{\prime})/R(s-1)$, respectively. It follows that

[TABLE]

where the first inequality is well-known (the $\ell_{1}$ operator norm bounds the $\ell_{2}$ operator norm for symmetric matrices) and the equality as well: the $\ell_{1}$ operator norm is the maximum absolute column sum. Since $\widetilde{S}$ was an arbitrary subset of size $2s$, the proof is complete. ∎