Improved Bound for the Nystrom's Method and its Application to Kernel Classification

TL;DR

This paper improves the theoretical error bounds for the Nyström method in kernel approximation, especially for matrices with skewed eigenvalues, and applies these results to enhance kernel classification efficiency.

Contribution

The paper introduces two novel approaches to tighten the error bounds of the Nyström method, particularly under large eigengaps and power-law eigenvalue decay, and applies these bounds to improve kernel classification.

Findings

Error bound improved from O(N/√m) to O(N/m^{1-ρ}) for large eigengaps.

Further improved to O(N/m^{p-1}) under power-law eigenvalues and incoherence.

Support vectors can be reduced to N^{2p/(p^2 - 1)} when eigenvalues follow a p-power law.

Abstract

We develop two approaches for analyzing the approximation error bound for the Nystr\"{o}m method, one based on the concentration inequality of integral operator, and one based on the compressive sensing theory. We show that the approximation error, measured in the spectral norm, can be improved from $O(N/\sqrt{m})$ to $O(N/m^{1 - \rho})$ in the case of large eigengap, where $N$ is the total number of data points, $m$ is the number of sampled data points, and $\rho \in (0, 1/2)$ is a positive constant that characterizes the eigengap. When the eigenvalues of the kernel matrix follow a $p$-power law, our analysis based on compressive sensing theory further improves the bound to $O(N/m^{p - 1})$ under an incoherence assumption, which explains why the Nystr\"{o}m method works well for kernel matrix with skewed eigenvalues. We present a kernel classification approach based on the Nystr\"{o}m method and derive its generalization performance using the improved bound. We show that when the eigenvalues of kernel matrix follow a $p$-power law, we can reduce the number of support vectors to $N^{2p/(p^2 - 1)}$, a number less than $N$ when $p > 1+\sqrt{2}$, without seriously sacrificing its generalization performance.