The gradient complexity of linear regression

TL;DR

This paper analyzes the computational complexity of linear regression and related problems, establishing tight bounds on the number of data access calls needed for polynomial accuracy using a matrix-vector product oracle.

Contribution

It provides a tight lower bound on the number of oracle calls required for linear regression, based on a reduction to eigenvalue estimation of Wishart matrices.

Findings

or polynomial accuracy, or randomized algorithms, ew key properties of Wishart matrices enable concise proofs.

The lower bound is or data access in linear regression.

Abstract

We investigate the computational complexity of several basic linear algebra primitives, including largest eigenvector computation and linear regression, in the computational model that allows access to the data via a matrix-vector product oracle. We show that for polynomial accuracy, $\Theta(d)$ calls to the oracle are necessary and sufficient even for a randomized algorithm. Our lower bound is based on a reduction to estimating the least eigenvalue of a random Wishart matrix. This simple distribution enables a concise proof, leveraging a few key properties of the random Wishart ensemble.