Learning Curves of Stochastic Gradient Descent in Kernel Regression

TL;DR

This paper analyzes the performance of stochastic gradient descent in kernel regression, revealing its optimal convergence rates and advantages over other methods, especially under model misspecification and specific step size schedules.

Contribution

It provides the first theoretical analysis showing SGD's optimal rates and advantages over iterative averaging in kernel regression with misspecified models.

Findings

SGD achieves min-max optimal rates across sample sizes.

SGD overcomes saturation phenomena common in ridge regression.

Exponential decay step size is key to SGD's success.

Abstract

This paper considers a canonical problem in kernel regression: how good are the model performances when it is trained by the popular online first-order algorithms, compared to the offline ones, such as ridge and ridgeless regression? In this paper, we analyze the foundational single-pass Stochastic Gradient Descent (SGD) in kernel regression under source condition where the optimal predictor can even not belong to the RKHS, i.e. the model is misspecified. Specifically, we focus on the inner product kernel over the sphere and characterize the exact orders of the excess risk curves under different scales of sample sizes $n$ concerning the input dimension $d$. Surprisingly, we show that SGD achieves min-max optimal rates up to constants among all the scales, without suffering the saturation, a prevalent phenomenon observed in (ridge) regression, except when the model is highly misspecified and the learning is in a final stage where $n\gg d^{\gamma}$ with any constant $\gamma >0$. The main reason for SGD to overcome the curse of saturation is the exponentially decaying step size schedule, a common practice in deep neural network training. As a byproduct, we provide the \emph{first} provable advantage of the scheme over the iterative averaging method in the common setting.