Bad estimation, good prediction: the Lasso in dense regimes

TL;DR

This paper shows that in dense high-dimensional models, the Lasso can produce accurate predictions even without sparsity, challenging the common interpretation of sparse models as biologically meaningful.

Contribution

The paper introduces a dense high-dimensional model where the Lasso achieves strong prediction bounds without assuming sparsity, supported by theoretical proofs and simulations.

Findings

Lasso prediction error decreases with more predictors in dense models

Sparse prediction rules can be highly accurate without sparse underlying signals

Caution is needed when interpreting sparse models as biologically meaningful

Abstract

For high-dimensional omics data, sparsity-inducing regularization methods such as the Lasso are widely used and often yield strong predictive performance, even in settings when the assumption of sparsity is likely violated. We demonstrate that under a specific dense model, namely the high-dimensional joint latent variable model, the Lasso produces sparse prediction rules with favorable prediction error bounds, even when the underlying regression coefficient vector is not sparse at all. We further argue that this model better represents many types of omics data than sparse linear regression models. We prove that the prediction bound under this model in fact decreases with increasing number of predictors, and confirm this through simulation examples. These results highlight the need for caution when interpreting sparse prediction rules, as strong prediction accuracy of a sparse prediction rule may not imply underlying biological significance of the individual predictors.