Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regression

TL;DR

This paper derives precise formulas for the learning curves of dot-product kernel regression across higher-order data and model scaling regimes, revealing complex behaviors and peaks in prediction error as data scales with dimension.

Contribution

It introduces exact formulas for test error, bias, and variance in higher-order asymptotics for kernel ridge regression, extending understanding beyond classical regimes.

Findings

Learning curves exhibit peaks at specific data scales, notably when m ≈ d^r/r!

Multiple descent phenomena occur at various scales in the higher-order asymptotics

The results unify and extend existing asymptotic theories for kernel regression.

Abstract

As modern machine learning models continue to advance the computational frontier, it has become increasingly important to develop precise estimates for expected performance improvements under different model and data scaling regimes. Currently, theoretical understanding of the learning curves that characterize how the prediction error depends on the number of samples is restricted to either large-sample asymptotics ($m\to\infty$) or, for certain simple data distributions, to the high-dimensional asymptotics in which the number of samples scales linearly with the dimension ($m\propto d$). There is a wide gulf between these two regimes, including all higher-order scaling relations $m\propto d^r$, which are the subject of the present paper. We focus on the problem of kernel ridge regression for dot-product kernels and present precise formulas for the mean of the test error, bias, and variance, for data drawn uniformly from the sphere with isotropic random labels in the $r$th-order asymptotic scaling regime $m\to\infty$ with $m/d^r$ held constant. We observe a peak in the learning curve whenever $m \approx d^r/r!$ for any integer $r$, leading to multiple sample-wise descent and nontrivial behavior at multiple scales.