The Lasso with general Gaussian designs with applications to hypothesis testing

TL;DR

This paper extends the theoretical understanding of the Lasso estimator to Gaussian correlated designs, providing non-asymptotic bounds and applications to hypothesis testing and confidence intervals.

Contribution

It generalizes the asymptotic characterization of the Lasso from i.i.d. Gaussian designs to correlated Gaussian designs with non-singular covariance matrices.

Findings

Established non-asymptotic bounds for the distributional difference between models.

Showed the necessity of degrees-of-freedom correction for valid confidence intervals.

Extended theoretical results to correlated Gaussian design matrices.

Abstract

The Lasso is a method for high-dimensional regression, which is now commonly used when the number of covariates $p$ is of the same order or larger than the number of observations $n$. Classical asymptotic normality theory does not apply to this model due to two fundamental reasons: $(1)$ The regularized risk is non-smooth; $(2)$ The distance between the estimator $\widehat{\boldsymbol{\theta}}$ and the true parameters vector $\boldsymbol{\theta}^*$ cannot be neglected. As a consequence, standard perturbative arguments that are the traditional basis for asymptotic normality fail. On the other hand, the Lasso estimator can be precisely characterized in the regime in which both $n$ and $p$ are large and $n/p$ is of order one. This characterization was first obtained in the case of Gaussian designs with i.i.d. covariates: here we generalize it to Gaussian correlated designs with non-singular covariance structure. This is expressed in terms of a simpler ``fixed-design'' model. We establish non-asymptotic bounds on the distance between the distribution of various quantities in the two models, which hold uniformly over signals $\boldsymbol{\theta}^*$ in a suitable sparsity class and over values of the regularization parameter. As an application, we study the distribution of the debiased Lasso and show that a degrees-of-freedom correction is necessary for computing valid confidence intervals.