Mixture model for designs in high dimensional regression and the LASSO

TL;DR

This paper introduces a Gaussian mixture model for the design matrix in high-dimensional regression, allowing the LASSO to perform well even when traditional incoherence conditions are not met, by focusing on cluster centers.

Contribution

It proposes a novel mixture model approach for design matrices, relaxing incoherence requirements and analyzing LASSO performance in clustered high-dimensional settings.

Findings

LASSO achieves similar accuracy under the mixture model as with incoherent designs.

Performance depends on the maximal variance within the mixture.

The model captures practical clustered structures in design matrices.

Abstract

The LASSO is a recent technique for variable selection in the regression model \bean y & = & X\beta + z, \eean where $X\in \R^{n\times p}$ and $z$ is a centered gaussian i.i.d. noise vector $\mathcal N(0,\sigma^2I)$. The LASSO has been proved to achieve remarkable properties such as exact support recovery of sparse vectors when the columns are sufficently incoherent and low prediction error under even less stringent conditions. However, many matrices do not satisfy small coherence in practical applications and the LASSO estimator may thus suffer from what is known as the slow rate regime. The goal of the present paper is to study the LASSO from a slightly different perspective by proposing a mixture model for the design matrix which is able to capture in a natural way the potentially clustered nature of the columns in many practical situations. In this model, the columns of the design matrix are drawn from a Gaussian mixture model. Instead of requiring incoherence for the design matrix $X$, we only require incoherence of the much smaller matrix of the mixture's centers. Our main result states that $X\beta$ can be estimated with the same precision as for incoherent designs except for a correction term depending on the maximal variance in the mixture model.