Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regime

TL;DR

This paper establishes that averaged stochastic gradient descent can achieve minimax optimal convergence rates for overparameterized neural networks in the NTK regime, leveraging the target function's complexity and RKHS properties.

Contribution

It provides the first convergence rate analysis showing optimal rates for averaged SGD in the NTK regime, connecting neural network learning to kernel methods.

Findings

Achieves minimax optimal convergence rate with global guarantees.

Shows NTK of ReLU networks can be learned at optimal rate.

Connects neural network training dynamics with kernel methods.

Abstract

We analyze the convergence of the averaged stochastic gradient descent for overparameterized two-layer neural networks for regression problems. It was recently found that a neural tangent kernel (NTK) plays an important role in showing the global convergence of gradient-based methods under the NTK regime, where the learning dynamics for overparameterized neural networks can be almost characterized by that for the associated reproducing kernel Hilbert space (RKHS). However, there is still room for a convergence rate analysis in the NTK regime. In this study, we show that the averaged stochastic gradient descent can achieve the minimax optimal convergence rate, with the global convergence guarantee, by exploiting the complexities of the target function and the RKHS associated with the NTK. Moreover, we show that the target function specified by the NTK of a ReLU network can be learned at the optimal convergence rate through a smooth approximation of a ReLU network under certain conditions.