The phase diagram of kernel interpolation in large dimensions

TL;DR

This paper characterizes the phase diagram of kernel interpolation in high dimensions, revealing when it achieves optimal, sub-optimal, or inconsistent generalization, thus shedding light on the benign overfitting phenomenon in neural networks.

Contribution

It provides a complete characterization of the bias and variance of kernel interpolation on the sphere in high dimensions, establishing the phase diagram in the $(s, \, \gamma)$-plane.

Findings

Identifies regions where kernel interpolation is minimax optimal.

Determines conditions leading to sub-optimal or inconsistent interpolation.

Provides exact order of bias and variance in large dimensions.

Abstract

The generalization ability of kernel interpolation in large dimensions (i.e., $n \asymp d^{\gamma}$ for some $\gamma>0$) might be one of the most interesting problems in the recent renaissance of kernel regression, since it may help us understand the 'benign overfitting phenomenon' reported in the neural networks literature. Focusing on the inner product kernel on the sphere, we fully characterized the exact order of both the variance and bias of large-dimensional kernel interpolation under various source conditions $s\geq 0$. Consequently, we obtained the $(s,\gamma)$-phase diagram of large-dimensional kernel interpolation, i.e., we determined the regions in $(s,\gamma)$-plane where the kernel interpolation is minimax optimal, sub-optimal and inconsistent.