Total stability of kernel methods

TL;DR

This paper establishes conditions for the stability of kernel methods in machine learning when the underlying probability measure, regularization, and kernel parameters vary slightly, extending classical robustness analysis.

Contribution

It provides new theoretical stability results for kernel methods considering simultaneous variations in measure, regularization, and kernel parameters, including for pairwise learning.

Findings

Kernel methods are stable under small simultaneous changes in measure, regularization, and kernel.

Results extend classical robustness to more general kernel learning scenarios.

Applicable to both standard and pairwise learning settings.

Abstract

Regularized empirical risk minimization using kernels and their corresponding reproducing kernel Hilbert spaces (RKHSs) plays an important role in machine learning. However, the actually used kernel often depends on one or on a few hyperparameters or the kernel is even data dependent in a much more complicated manner. Examples are Gaussian RBF kernels, kernel learning, and hierarchical Gaussian kernels which were recently proposed for deep learning. Therefore, the actually used kernel is often computed by a grid search or in an iterative manner and can often only be considered as an approximation to the "ideal" or "optimal" kernel. The paper gives conditions under which classical kernel based methods based on a convex Lipschitz loss function and on a bounded and smooth kernel are stable, if the probability measure $P$, the regularization parameter $\lambda$, and the kernel $k$ may slightly change in a simultaneous manner. Similar results are also given for pairwise learning. Therefore, the topic of this paper is somewhat more general than in classical robust statistics, where usually only the influence of small perturbations of the probability measure $P$ on the estimated function is considered.