Robust, randomized preconditioning for kernel ridge regression

TL;DR

This paper introduces two robust randomized preconditioning methods for kernel ridge regression that significantly improve computational efficiency for large datasets, with guarantees under specific eigenvalue decay conditions.

Contribution

It presents RPCholesky and KRILL preconditioning techniques that enable efficient, accurate solutions to large-scale KRR problems with theoretical guarantees.

Findings

RPCholesky solves full-data KRR in $O(N^2)$ operations.

KRILL efficiently handles KRR with selected data centers.

Both methods are practical for large datasets.

Abstract

This paper investigates preconditioned conjugate gradient techniques for solving kernel ridge regression (KRR) problems with a medium to large number of data points ($10^4 \leq N \leq 10^7$), and it describes two methods with the strongest guarantees available. The first method, RPCholesky preconditioning, accurately solves the full-data KRR problem in $O(N^2)$ arithmetic operations, assuming sufficiently rapid polynomial decay of the kernel matrix eigenvalues. The second method, KRILL preconditioning, offers an accurate solution to a restricted version of the KRR problem involving $k \ll N$ selected data centers at a cost of $O((N + k^2) k \log k)$ operations. The proposed methods efficiently solve a range of KRR problems, making them well-suited for practical applications.