Benefit of Interpolation in Nearest Neighbor Algorithms

TL;DR

This paper demonstrates that in nearest neighbor algorithms, a mild degree of data interpolation can improve prediction accuracy and stability, challenging the notion that zero training error always harms generalization.

Contribution

It introduces a class of interpolated weighting schemes in nearest neighbors, revealing a U-shaped performance curve and showing that mild interpolation can enhance predictive performance.

Findings

Mild data interpolation improves NN prediction accuracy.

Zero training error does not necessarily harm generalization.

Universality of results across distance measures and corrupted data.

Abstract

In some studies \citep[e.g.,][]{zhang2016understanding} of deep learning, it is observed that over-parametrized deep neural networks achieve a small testing error even when the training error is almost zero. Despite numerous works towards understanding this so-called "double descent" phenomenon \citep[e.g.,][]{belkin2018reconciling,belkin2019two}, in this paper, we turn into another way to enforce zero training error (without over-parametrization) through a data interpolation mechanism. Specifically, we consider a class of interpolated weighting schemes in the nearest neighbors (NN) algorithms. By carefully characterizing the multiplicative constant in the statistical risk, we reveal a U-shaped performance curve for the level of data interpolation in both classification and regression setups. This sharpens the existing result \citep{belkin2018does} that zero training error does not necessarily jeopardize predictive performances and claims a counter-intuitive result that a mild degree of data interpolation actually {\em strictly} improve the prediction performance and statistical stability over those of the (un-interpolated) $k$-NN algorithm. In the end, the universality of our results, such as change of distance measure and corrupted testing data, will also be discussed.