Revisiting Approximate Leverage Score Sketching for Matrix Least Squares

TL;DR

This paper analyzes the effectiveness of approximate leverage score sampling for matrix least squares problems, providing theoretical bounds and improvements over previous results.

Contribution

It derives first-principles bounds for leverage score sketching, clarifies their relation to prior bounds, and introduces a hybrid sampling scheme to reduce sample complexity.

Findings

Approximate leverage score sampling requires $4r/(eta heta ext{-} ext{epsilon})$ samples for accurate solutions.

The quality measure $eta$ influences the sample size needed, with $eta=1$ for exact leverage scores.

Hybrid deterministic-random sampling reduces sample complexity when large approximate leverage scores are present.

Abstract

We revisit the problem of sketching using approximate leverage scores for matrix least squares problems of the form $\| AX - B \|_F^2$ where the design matrix $A \in \mathbb{R}^{N \times r}$ is tall and skinny with $N \gg r$. We derive the theoretical results from first principles and clarify the relation to previously stated bounds, improving some constants along the way. One can characterize the utility of a sketching scheme according to the number of samples it needs for an $\varepsilon$-accurate solution with high probability. Assuming $\varepsilon$ is suitably small, we will show that approximate leverage score sampling requires $4r/(\beta\delta\varepsilon)$ samples, where $\delta$ is the failure probability and $\beta \in (0,1]$ is a measure of the quality of the approximate leverage scores such that $\beta=1$ corresponds to using exact leverage scores. In cases where a few approximate leverage scores are very large (summing to $p_{\rm det}$), we also show that using a hybrid deterministic and random sampling scheme reduces the required number of samples by a factor of $1/(1-p_{\rm det})$.