Just Interpolate: Kernel "Ridgeless" Regression Can Generalize

TL;DR

This paper investigates how kernel ridgeless regression can generalize well without explicit regularization, due to implicit regularization effects from data geometry, kernel curvature, and high dimensionality, supported by theoretical bounds and experiments.

Contribution

It identifies and explains the implicit regularization mechanism enabling kernel ridgeless regression to generalize, with theoretical bounds and empirical evidence.

Findings

Implicit regularization arises from data geometry and kernel properties.

Theoretical upper bounds on out-of-sample error are derived.

Experimental results on MNIST support the phenomenon.

Abstract

In the absence of explicit regularization, Kernel "Ridgeless" Regression with nonlinear kernels has the potential to fit the training data perfectly. It has been observed empirically, however, that such interpolated solutions can still generalize well on test data. We isolate a phenomenon of implicit regularization for minimum-norm interpolated solutions which is due to a combination of high dimensionality of the input data, curvature of the kernel function, and favorable geometric properties of the data such as an eigenvalue decay of the empirical covariance and kernel matrices. In addition to deriving a data-dependent upper bound on the out-of-sample error, we present experimental evidence suggesting that the phenomenon occurs in the MNIST dataset.