Kernel PCA with the Nystr\"om method

TL;DR

This paper introduces a scalable version of kernel PCA using the Nyström method, providing theoretical accuracy guarantees and demonstrating its effectiveness through experiments and an application to kernel principal component regression.

Contribution

It derives kernel PCA with the Nyström method, offering a scalable alternative with statistical accuracy bounds and practical applications.

Findings

The method is scalable to large datasets.

Finite-sample confidence bounds are established.

Experimental results validate the approach.

Abstract

The Nystr\"om method is one of the most popular techniques for improving the scalability of kernel methods. However, it has not yet been derived for kernel PCA in line with classical PCA. In this paper we derive kernel PCA with the Nystr\"om method, thereby providing one of the few available options to make kernel PCA scalable. We further study its statistical accuracy through a finite-sample confidence bound on the empirical reconstruction error compared to the full method. The behaviours of the method and bound are illustrated through computer experiments on multiple real-world datasets. As an application of the method we present kernel principal component regression with the Nystr\"om method, as an alternative to Nystr\"om kernel ridge regression for efficient regularized regression with kernels.