Phase transition and higher order analysis of $L_q$ regularization under dependence

TL;DR

This paper analyzes the phase transition and higher order behavior of $L_q$ regularization methods for sparse signal recovery under dependence, revealing how covariance affects performance especially at $q=1$ (LASSO).

Contribution

It provides the first higher-order asymptotic analysis of $L_q$ regularized least squares under dependence, generalizing previous Gaussian design results and clarifying the role of covariance structure.

Findings

The first dominant term in risk does not depend on covariance for $0 extless q extless 1$ and $1 extless q extless 2$.

Covariance influences phase transition only at $q=1$ (LASSO).

Explicit formulas for second dominant term in risk expansion are derived.

Abstract

We study the problem of estimating a $k$-sparse signal ${\mbox{$\beta$}}_0\in{\bf R}^p$ from a set of noisy observations ${\bf y}\in{\bf R}^n$ under the model ${\bf y}={\bf X}{\mbox{$\beta$}}+{\bf w}$, where ${\bf X}\in{\bf R}^{n\times p}$ is the measurement matrix the row of which is drawn from distribution $N(0,{\mbox{$\Sigma$}})$. We consider the class of $L_q$-regularized least squares (LQLS) given by the formulation $\hat{\mbox{$\beta$}}(\lambda,q)=\text{argmin}_{{\mbox{$\beta$}}\in{\bf R}^p}\frac{1}{2}\|{\bf y}-{\bf X}{\mbox{$\beta$}}\|^2_2+\lambda\|{\mbox{$\beta$}}\|_q^q$, where $\|\cdot\|_q$ $(0\le q\le 2)$ denotes the $L_q$-norm. In the setting $p,n,k\rightarrow\infty$ with fixed $k/p=\epsilon$ and $n/p=\delta$, we derive the asymptotic risk of $\hat{\mbox{$\beta$}}(\lambda,q)$ for arbitrary covariance matrix ${\mbox{$\Sigma$}}$ which generalizes the existing results for standard Gaussian design, i.e. $X_{ij}\overset{i.i.d}{\sim}N(0,1)$. We perform a higher-order analysis for LQLS in the small-error regime in which the first dominant term can be used to determine the phase transition behavior of LQLS. Our results show that the first dominant term does not depend on the covariance structure of ${\mbox{$\Sigma$}}$ in the cases $0\le q< 1$ and $1< q\le 2$ which indicates that the correlations among predictors only affect the phase transition curve in the case $q=1$ a.k.a. LASSO. To study the influence of the covariance structure of ${\mbox{$\Sigma$}}$ on the performance of LQLS in the cases $0\le q< 1$ and $1<q\le 2$, we derive the explicit formulas for the second dominant term in the expansion of the asymptotic risk in terms of small error. Extensive computational experiments confirm that our analytical predictions are consistent with numerical results.