Preconditioning without a preconditioner: faster ridge-regression and Gaussian sampling with randomized block Krylov subspace methods

TL;DR

This paper introduces a randomized block Krylov subspace method that accelerates solving positive-definite linear systems and improves data science tasks like ridge regression and Gaussian sampling without explicit preconditioning.

Contribution

It presents a novel randomized algorithm that outperforms traditional preconditioned methods and provides theoretical guarantees for Nyström-based preconditioning variants.

Findings

Outperforms preconditioned conjugate gradient in speed

Enables faster ridge regression regularization path computation

Generates high-dimensional Gaussian samples efficiently

Abstract

We describe a randomized variant of the block conjugate gradient method for solving a single positive-definite linear system of equations. Our method provably outperforms preconditioned conjugate gradient with a broad-class of Nystr\"om-based preconditioners, without ever explicitly constructing a preconditioner. In analyzing our algorithm, we derive theoretical guarantees for new variants of Nystr\"om preconditioned conjugate gradient which may be of separate interest. We also describe how our approach yields state-of-the-art algorithms for key data-science tasks such as computing the entire ridge regression regularization path and generating multiple independent samples from a high-dimensional Gaussian distribution.