Overfitting Behaviour of Gaussian Kernel Ridgeless Regression: Varying Bandwidth or Dimensionality

TL;DR

This paper investigates how the overfitting behavior of Gaussian kernel ridgeless regression changes with bandwidth and input dimension, revealing limitations in consistency and introducing benign overfitting scenarios.

Contribution

It provides the first example of benign overfitting with Gaussian kernels under sub-polynomial dimension scaling and characterizes overfitting across different dimensions and bandwidths.

Findings

Ridgeless solutions are never consistent at fixed dimensions.

Large noise leads to worse performance than null predictor.

Benign overfitting occurs with sub-polynomial dimension growth.

Abstract

We consider the overfitting behavior of minimum norm interpolating solutions of Gaussian kernel ridge regression (i.e. kernel ridgeless regression), when the bandwidth or input dimension varies with the sample size. For fixed dimensions, we show that even with varying or tuned bandwidth, the ridgeless solution is never consistent and, at least with large enough noise, always worse than the null predictor. For increasing dimension, we give a generic characterization of the overfitting behavior for any scaling of the dimension with sample size. We use this to provide the first example of benign overfitting using the Gaussian kernel with sub-polynomial scaling dimension. All our results are under the Gaussian universality ansatz and the (non-rigorous) risk predictions in terms of the kernel eigenstructure.