High-dimensional learning dynamics of multi-pass Stochastic Gradient Descent in multi-index models

TL;DR

This paper provides an exact asymptotic analysis of the learning dynamics of multi-pass mini-batch SGD in high-dimensional multi-index models, revealing the effects of batch size and learning rate scaling.

Contribution

It introduces a novel dynamical mean-field framework and scalar Poisson jump process to characterize SGD dynamics in high dimensions, extending existing models.

Findings

SGD dynamics are invariant across batch size scalings within [0,1)

SGD, SME, and gradient flow have distinct dynamics under certain scalings

The analysis recovers known results for gradient flow and online SGD in specific limits

Abstract

We study the learning dynamics of a multi-pass, mini-batch Stochastic Gradient Descent (SGD) procedure for empirical risk minimization in high-dimensional multi-index models with isotropic random data. In an asymptotic regime where the sample size $n$ and data dimension $d$ increase proportionally, for any sub-linear batch size $\kappa \asymp n^\alpha$ where $\alpha \in [0,1)$, and for a commensurate ``critical'' scaling of the learning rate, we provide an asymptotically exact characterization of the coordinate-wise dynamics of SGD. This characterization takes the form of a system of dynamical mean-field equations, driven by a scalar Poisson jump process that represents the asymptotic limit of SGD sampling noise. We develop an analogous characterization of the Stochastic Modified Equation (SME) which provides a Gaussian diffusion approximation to SGD. Our analyses imply that the limiting dynamics for SGD are the same for any batch size scaling $\alpha \in [0,1)$, and that under a commensurate scaling of the learning rate, dynamics of SGD, SME, and gradient flow are mutually distinct, with those of SGD and SME coinciding in the special case of a linear model. We recover a known dynamical mean-field characterization of gradient flow in a limit of small learning rate, and of one-pass/online SGD in a limit of increasing sample size $n/d \to \infty$.