Fast Sparse Least-Squares Regression with Non-Asymptotic Guarantees

TL;DR

This paper introduces a fast approximation method for large-scale high-dimensional sparse least-squares regression using Johnson-Lindenstrauss transforms, providing non-asymptotic error bounds and insights into sample complexity and regularization.

Contribution

It proposes a novel approach applying JL transforms to data for efficient sparse regression with theoretical non-asymptotic guarantees and analysis of regularizers.

Findings

Establishes non-asymptotic error bounds for elastic net and L1 regularized models.

Provides sample complexity analysis under JL transforms.

Derives error bounds for the Dantzig selector with JL transforms.

Abstract

In this paper, we study a fast approximation method for {\it large-scale high-dimensional} sparse least-squares regression problem by exploiting the Johnson-Lindenstrauss (JL) transforms, which embed a set of high-dimensional vectors into a low-dimensional space. In particular, we propose to apply the JL transforms to the data matrix and the target vector and then to solve a sparse least-squares problem on the compressed data with a {\it slightly larger regularization parameter}. Theoretically, we establish the optimization error bound of the learned model for two different sparsity-inducing regularizers, i.e., the elastic net and the $\ell_1$ norm. Compared with previous relevant work, our analysis is {\it non-asymptotic and exhibits more insights} on the bound, the sample complexity and the regularization. As an illustration, we also provide an error bound of the {\it Dantzig selector} under JL transforms.