Sharp Analysis of Sketch-and-Project Methods via a Connection to Randomized Singular Value Decomposition

TL;DR

This paper provides a theoretical analysis of sketch-and-project methods, revealing how convergence improves with sketch size and spectral properties, and introduces sparse sketching matrices that maintain efficiency and robustness.

Contribution

It establishes the first sharp convergence guarantees for sketch-and-project methods, linking their performance to randomized SVD and enabling sparse, efficient sketches.

Findings

Convergence rate improves linearly with sketch size.

Sparse sketches do not impair convergence rates.

Spectral decay enhances convergence speed.

Abstract

Sketch-and-project is a framework which unifies many known iterative methods for solving linear systems and their variants, as well as further extensions to non-linear optimization problems. It includes popular methods such as randomized Kaczmarz, coordinate descent, variants of the Newton method in convex optimization, and others. In this paper, we develop a theoretical framework for obtaining sharp guarantees on the convergence rate of sketch-and-project methods. Our approach is the first to: (1) show that the convergence rate improves at least linearly with the sketch size, and even faster when the data matrix exhibits certain spectral decays; and (2) allow for sparse sketching matrices, which are more efficient than dense sketches and more robust than sub-sampling methods. In particular, our results explain an observed phenomenon that a radical sparsification of the sketching matrix does not affect the per iteration convergence rate of sketch-and-project. To obtain our results, we develop new non-asymptotic spectral bounds for the expected sketched projection matrix, which are of independent interest; and we establish a connection between the convergence rates of iterative sketch-and-project solvers and the approximation error of randomized singular value decomposition, which is a widely used one-shot sketching algorithm for low-rank approximation. Our experiments support the theory and demonstrate that even extremely sparse sketches exhibit the convergence properties predicted by our framework.