Random Matrix Theory of Early-Stopped Gradient Flow: A Transient BBP Scenario

TL;DR

This paper models the transient detection of signals in early-stopped gradient flow using random matrix theory, revealing how anisotropy and noise influence spectral transitions and early stopping.

Contribution

It introduces an analytically tractable random matrix model capturing the transient spectral phenomena in early-stopped gradient flow with anisotropic input covariance.

Findings

Transient eigenvalue separation indicates signal detection before overfitting.

Derived full time-dependent bulk spectrum using a 2x2 Dyson equation.

Mapped phase diagrams showing conditions for signal emergence and reabsorption.

Abstract

Empirical studies of trained models often report a transient regime in which signal is detectable in a finite gradient descent time window before overfitting dominates. We provide an analytically tractable random-matrix model that reproduces this phenomenon for gradient flow in a linear teacher--student setting. In this framework, learning occurs when an isolated eigenvalue separates from a noisy bulk, before eventually disappearing in the overfitting regime. The key ingredient is anisotropy in the input covariance, which induces fast and slow directions in the learning dynamics. In a two-block covariance model, we derive the full time-dependent bulk spectrum of the symmetrized weight matrix through a $2\times 2$ Dyson equation, and we obtain an explicit outlier condition for a rank-one teacher via a rank-two determinant formula. This yields a transient Baik-Ben Arous-P\'ech\'e (BBP) transition: depending on signal strength and covariance anisotropy, the teacher spike may never emerge, emerge and persist, or emerge only during an intermediate time interval before being reabsorbed into the bulk. We map the corresponding phase diagrams and validate the theory against finite-size simulations. Our results provide a minimal solvable mechanism for early stopping as a transient spectral effect driven by anisotropy and noise.