On prediction with the LASSO when the design is not incoherent

TL;DR

This paper introduces a new index to evaluate how suitable a design matrix is for LASSO prediction, enabling bounds without traditional incoherence assumptions.

Contribution

It proposes a novel index for assessing design matrices' favorability for LASSO and demonstrates bounds applicable to arbitrary matrices with restricted non-singularity.

Findings

New index effectively measures design matrix suitability for LASSO.

Appending random matrices extends prediction bounds to general designs.

Prediction bounds hold under minimal assumptions, broadening applicability.

Abstract

The LASSO estimator is an $\ell_1$-norm penalized least-squares estimator, which was introduced for variable selection in the linear model. When the design matrix satisfies, e.g. the Restricted Isometry Property, or has a small coherence index, the LASSO estimator has been proved to recover, with high probability, the support and sign pattern of sufficiently sparse regression vectors. Under similar assumptions, the LASSO satisfies adaptive prediction bounds in various norms. The present note provides a prediction bound based on a new index for measuring how favorable is a design matrix for the LASSO estimator. We study the behavior of our new index for matrices with independent random columns uniformly drawn on the unit sphere. Using the simple trick of appending such a random matrix (with the right number of columns) to a given design matrix, we show that a prediction bound similar to \cite[Theorem 2.1]{CandesPlan:AnnStat09} holds without any constraint on the design matrix, other than restricted non-singularity.