Beyond Benign Overfitting in Nadaraya-Watson Interpolators

TL;DR

This paper investigates the generalization behavior of the classic Nadaraya-Watson interpolator, revealing multiple overfitting regimes and providing insights into hyperparameter tuning and data dimension estimation.

Contribution

It offers a theoretical analysis of the Nadaraya-Watson estimator's overfitting behaviors and demonstrates these phenomena through numerical experiments.

Findings

Multiple overfitting regimes identified, from catastrophic to benign.

Over-estimating data dimension is less harmful than under-estimating.

Numerical experiments confirm theoretical predictions.

Abstract

In recent years, there has been much interest in understanding the generalization behavior of interpolating predictors, which overfit on noisy training data. Whereas standard analyses are concerned with whether a method is consistent or not, recent observations have shown that even inconsistent predictors can generalize well. In this work, we revisit the classic interpolating Nadaraya-Watson (NW) estimator (also known as Shepard's method), and study its generalization capabilities through this modern viewpoint. In particular, by varying a single bandwidth-like hyperparameter, we prove the existence of multiple overfitting behaviors, ranging non-monotonically from catastrophic, through benign, to tempered. Our results highlight how even classical interpolating methods can exhibit intricate generalization behaviors. In addition, for the purpose of tuning the hyperparameter, the results suggest that over-estimating the intrinsic dimension of the data is less harmful than under-estimating it. Numerical experiments complement our theory, demonstrating the same phenomena.