Sharp analysis of sketched least squares and randomized low-rank approximation

TL;DR

This paper rigorously analyzes the optimal random embeddings for sketch-and-solve least squares and randomized SVD, establishing minimax optimality and providing sharp error bounds, supported by empirical evidence of universality.

Contribution

It proves the minimax optimality of specific random embeddings for these algorithms and derives the sharpest possible error bounds, advancing theoretical understanding.

Findings

Random orthonormal matrices are minimax optimal for sketch-and-solve.

Rotation-invariant embeddings are minimax optimal for randomized SVD.

Empirical results show different random embeddings achieve similar accuracy in practice.

Abstract

Two widely used randomized algorithms are the sketch-and-solve method for least-squares regression and the randomized SVD for low-rank approximation. These algorithms apply a random embedding to compress a target matrix, and they perform computations on the compressed matrix to save computational cost. This paper asks, what is the optimal random embedding in these algorithms? Also, what is the sharpest possible error bound for the optimal embedding? The paper proves that a random orthonormal matrix is minimax optimal for the sketch-and-solve algorithm while any rotation-invariant embedding is minimax optimal for the randomized SVD. Following these results, the paper obtains the best possible error bounds for sketched least-squares and the randomized SVD. Last, empirical experiments provide evidence of universality phenomena, in which several random embeddings lead to similar accuracy to the optimal embeddings in practice.