Optimal Rate of Kernel Regression in Large Dimensions

TL;DR

This paper analyzes the optimal convergence rates of kernel regression in high-dimensional settings, revealing new phenomena like multiple descent and periodic plateau behaviors, with implications for neural networks and the neural tangent kernel.

Contribution

It introduces a general framework to characterize minimax bounds for kernel regression in large dimensions and identifies the precise rates and phenomena across different sample size regimes.

Findings

Minimax rate of $n^{-1/2}$ for $ ext{γ} = 2, 4, 6, 8, ext{etc.}$

Discovery of multiple descent and periodic plateau behaviors in the optimal rate curve

Explicit description of the optimal rate curve for the neural tangent kernel (NTK)

Abstract

We perform a study on kernel regression for large-dimensional data (where the sample size $n$ is polynomially depending on the dimension $d$ of the samples, i.e., $n\asymp d^{\gamma}$ for some $\gamma >0$ ). We first build a general tool to characterize the upper bound and the minimax lower bound of kernel regression for large dimensional data through the Mendelson complexity $\varepsilon_{n}^{2}$ and the metric entropy $\bar{\varepsilon}_{n}^{2}$ respectively. When the target function falls into the RKHS associated with a (general) inner product model defined on $\mathbb{S}^{d}$, we utilize the new tool to show that the minimax rate of the excess risk of kernel regression is $n^{-1/2}$ when $n\asymp d^{\gamma}$ for $\gamma =2, 4, 6, 8, \cdots$. We then further determine the optimal rate of the excess risk of kernel regression for all the $\gamma>0$ and find that the curve of optimal rate varying along $\gamma$ exhibits several new phenomena including the multiple descent behavior and the periodic plateau behavior. As an application, For the neural tangent kernel (NTK), we also provide a similar explicit description of the curve of optimal rate. As a direct corollary, we know these claims hold for wide neural networks as well.