Non-negative least squares for high-dimensional linear models: consistency and sparse recovery without regularization

TL;DR

This paper demonstrates that non-negative least squares can achieve consistent estimation and support recovery in high-dimensional linear models without regularization, under certain design matrix conditions, offering a simpler alternative to regularized methods.

Contribution

It shows that NNLS can be as effective as lasso in high-dimensional settings without regularization, given specific properties of the design matrix.

Findings

NNLS performs comparably to lasso in prediction and estimation.

NNLS may have better support recovery with thresholding.

NNLS does not require tuning a regularization parameter.

Abstract

Least squares fitting is in general not useful for high-dimensional linear models, in which the number of predictors is of the same or even larger order of magnitude than the number of samples. Theory developed in recent years has coined a paradigm according to which sparsity-promoting regularization is regarded as a necessity in such setting. Deviating from this paradigm, we show that non-negativity constraints on the regression coefficients may be similarly effective as explicit regularization if the design matrix has additional properties, which are met in several applications of non-negative least squares (NNLS). We show that for these designs, the performance of NNLS with regard to prediction and estimation is comparable to that of the lasso. We argue further that in specific cases, NNLS may have a better $\ell_{\infty}$-rate in estimation and hence also advantages with respect to support recovery when combined with thresholding. From a practical point of view, NNLS does not depend on a regularization parameter and is hence easier to use.