Reduced-Set Kernel Principal Components Analysis for Improving the Training and Execution Speed of Kernel Machines

TL;DR

This paper introduces Reduced Set KPCA (RSKPCA), a method that speeds up kernel PCA by selecting representative data points, reducing computational costs while maintaining accuracy, and outperforming existing approximation techniques.

Contribution

The paper proposes RSKPCA, a novel, theoretically grounded approach for faster kernel PCA through sample reduction and density-based data substitution.

Findings

RSKPCA speeds up KPCA by up to ten times.

It maintains spectral and operator accuracy comparable to full KPCA.

Outperforms Nystrom and density-weighted Nystrom methods in experiments.

Abstract

This paper presents a practical, and theoretically well-founded, approach to improve the speed of kernel manifold learning algorithms relying on spectral decomposition. Utilizing recent insights in kernel smoothing and learning with integral operators, we propose Reduced Set KPCA (RSKPCA), which also suggests an easy-to-implement method to remove or replace samples with minimal effect on the empirical operator. A simple data point selection procedure is given to generate a substitute density for the data, with accuracy that is governed by a user-tunable parameter . The effect of the approximation on the quality of the KPCA solution, in terms of spectral and operator errors, can be shown directly in terms of the density estimate error and as a function of the parameter . We show in experiments that RSKPCA can improve both training and evaluation time of KPCA by up to an order of magnitude, and compares favorably to the widely-used Nystrom and density-weighted Nystrom methods.