Asymptotics for Sketching in Least Squares Regression

TL;DR

This paper analyzes the asymptotic behavior of sketching methods in least squares regression, revealing fundamental differences between methods like SRHT and Gaussian projections, and providing theoretical limits on their accuracy loss.

Contribution

It provides a fine-grained asymptotic analysis distinguishing the performance of different sketching methods in least squares regression.

Findings

SRHT outperforms Gaussian projections in asymptotic accuracy.

Theoretical limits of estimation and test error are established.

Results are validated on real and synthetic datasets.

Abstract

We consider a least squares regression problem where the data has been generated from a linear model, and we are interested to learn the unknown regression parameters. We consider "sketch-and-solve" methods that randomly project the data first, and do regression after. Previous works have analyzed the statistical and computational performance of such methods. However, the existing analysis is not fine-grained enough to show the fundamental differences between various methods, such as the Subsampled Randomized Hadamard Transform (SRHT) and Gaussian projections. In this paper, we make progress on this problem, working in an asymptotic framework where the number of datapoints and dimension of features goes to infinity. We find the limits of the accuracy loss (for estimation and test error) incurred by popular sketching methods. We show separation between different methods, so that SRHT is better than Gaussian projections. Our theoretical results are verified on both real and synthetic data. The analysis of SRHT relies on novel methods from random matrix theory that may be of independent interest.