Less is More: Nystr\"om Computational Regularization

TL;DR

This paper analyzes Nyström subsampling methods for large-scale kernel learning, providing theoretical bounds and demonstrating that appropriate subsampling acts as both a regularizer and a computational efficiency tool, with strong empirical results.

Contribution

It offers new theoretical learning bounds for Nyström methods and introduces an incremental variant that balances regularization and computation effectively.

Findings

Achieves optimal learning bounds with proper subsampling

Demonstrates state-of-the-art performance on large datasets

Proposes a simple incremental Nyström regularization method

Abstract

We study Nystr\"om type subsampling approaches to large scale kernel methods, and prove learning bounds in the statistical learning setting, where random sampling and high probability estimates are considered. In particular, we prove that these approaches can achieve optimal learning bounds, provided the subsampling level is suitably chosen. These results suggest a simple incremental variant of Nystr\"om Kernel Regularized Least Squares, where the subsampling level implements a form of computational regularization, in the sense that it controls at the same time regularization and computations. Extensive experimental analysis shows that the considered approach achieves state of the art performances on benchmark large scale datasets.