Faster Low-Rank Approximation and Kernel Ridge Regression via the Block-Nystr\"om Method

TL;DR

This paper introduces Block-Nystr"om, a novel algorithm that enhances low-rank matrix approximation and kernel ridge regression efficiency by incorporating block-diagonal structures, improving computational costs and approximation quality.

Contribution

The paper proposes Block-Nystr"om, a new method that combines multiple smaller approximations to achieve better spectral tail estimates and computational efficiency in kernel methods.

Findings

Block-Nystr"om reduces computational cost significantly.

It provides stronger spectral tail estimates than traditional methods.

The method improves kernel ridge regression performance.

Abstract

The Nystr\"om method is a popular low-rank approximation technique for large matrices that arise in kernel methods and convex optimization. Yet, when the data exhibits heavy-tailed spectral decay, the effective dimension of the problem often becomes so large that even the Nystr\"om method may be outside of our computational budget. To address this, we propose Block-Nystr\"om, an algorithm that injects a block-diagonal structure into the Nystr\"om method, thereby significantly reducing its computational cost while recovering strong approximation guarantees. We show that Block-Nystr\"om can be used to construct improved preconditioners for second-order optimization, as well as to efficiently solve kernel ridge regression for statistical learning over Hilbert spaces. Our key technical insight is that, within the same computational budget, combining several smaller Nystr\"om approximations leads to stronger tail estimates of the input spectrum than using one larger approximation. Along the way, we provide a novel recursive preconditioning scheme for efficiently inverting the Block-Nystr\"om matrix, and provide new statistical learning bounds for a broad class of approximate kernel ridge regression solvers.