Kernel ridge vs. principal component regression: minimax bounds and adaptability of regularization operators

TL;DR

This paper provides a comprehensive comparison of regularization techniques like ridge and principal component regression in kernel methods, deriving risk bounds and demonstrating their adaptability and minimax optimality in nonparametric regression.

Contribution

It introduces explicit finite-sample risk bounds for regularization estimators that incorporate function space structure, regularity, and adaptability, highlighting differences between ridge and principal component regression.

Findings

Risk bounds match minimax rates in various settings.

Some regularization methods are more adaptable to function regularity.

Kernel ridge regression and principal component regression differ significantly in adaptability.

Abstract

Regularization is an essential element of virtually all kernel methods for nonparametric regression problems. A critical factor in the effectiveness of a given kernel method is the type of regularization that is employed. This article compares and contrasts members from a general class of regularization techniques, which notably includes ridge regression and principal component regression. We derive an explicit finite-sample risk bound for regularization-based estimators that simultaneously accounts for (i) the structure of the ambient function space, (ii) the regularity of the true regression function, and (iii) the adaptability (or qualification) of the regularization. A simple consequence of this upper bound is that the risk of the regularization-based estimators matches the minimax rate in a variety of settings. The general bound also illustrates how some regularization techniques are more adaptable than others to favorable regularity properties that the true regression function may possess. This, in particular, demonstrates a striking difference between kernel ridge regression and kernel principal component regression. Our theoretical results are supported by numerical experiments.