Spectral Edge Dynamics of Training Trajectories: Signal--Noise Geometry Across Scales

TL;DR

This paper introduces Spectral Edge Dynamics (SED), a method to analyze training trajectories of large models by identifying a spectral boundary that separates coherent optimization directions from noise, revealing universal patterns across scales.

Contribution

The paper presents SED, a novel spectral analysis framework that uncovers universal geometric patterns in training trajectories and relates spectral geometry to model complexity and generalization signals.

Findings

Spectral edge exhibits a universal three-phase pattern during training.

Effective signal rank adapts to task complexity, e.g., 2 at 51M and 3 at 124M parameters.

Spectral gap preservation enables analysis of models at arbitrary scale.

Abstract

Despite hundreds of millions of parameters, transformer training trajectories evolve within only a few coherent directions. We introduce Spectral Edge Dynamics (SED) to quantify this structure: a rolling-window SVD of parameter updates reveals a sharp boundary -- the spectral edge -- between coherent optimization directions and stochastic noise, identified via the maximum consecutive singular value ratio $\sigma_k / \sigma_{k+1}$. Across a 51M-parameter TinyStories model (4 seeds) and GPT-2 124M under distribution shift, the spectral edge exhibits a universal three-phase pattern (rise, plateau, collapse). The effective signal rank adapts to task complexity ($k^* = 2$ at 51M, $k^* = 3$ at 124M), and the directional coupling between spectral geometry and validation loss reverses with window size -- a lag flip reflecting the timescale of trajectory integration. Johnson--Lindenstrauss projection to $d = 10W$ dimensions (e.g., $d = 100$ for $W = 10$) preserves the spectral gap within $5.7\%$, making the framework applicable to models of arbitrary scale. In companion work, the same spectral geometry provides early-warning signals of grokking -- predicting generalization $600$--$1{,}700$ steps before it occurs across modular arithmetic, Dyck languages, and the SCAN benchmark.