Grokking as a Variance-Limited Phase Transition: Spectral Gating and the Epsilon-Stability Threshold

TL;DR

This paper models grokking as a phase transition governed by spectral gating and variance regulation in AdamW optimizer dynamics, revealing the conditions under which generalization emerges long after training convergence.

Contribution

It introduces a spectral gating mechanism explaining grokking, highlighting the role of variance and stability thresholds in optimizer dynamics, and challenges existing hypotheses about flat minima.

Findings

Grokking occurs when gradient variance surpasses a stability threshold.

AdamW's anisotropic noise directs generalization in a spectral gating regime.

Three complexity regimes influence the learning dynamics and grokking onset.

Abstract

Standard optimization theories struggle to explain grokking, where generalization occurs long after training convergence. While geometric studies attribute this to slow drift, they often overlook the interaction between the optimizer's noise structure and landscape curvature. This work analyzes AdamW dynamics on modular arithmetic tasks, revealing a ``Spectral Gating'' mechanism that regulates the transition from memorization to generalization. We find that AdamW operates as a variance-gated stochastic system. Grokking is constrained by a stability condition: the generalizing solution resides in a sharp basin ($\lambda_{max}^H$) initially inaccessible under low-variance regimes. The ``delayed'' phase represents the accumulation of gradient variance required to lift the effective stability ceiling, permitting entry into this sharp manifold. Our ablation studies identify three complexity regimes: (1) \textbf{Capacity Collapse} ($P < 23$), where rank-deficiency prevents structural learning; (2) \textbf{The Variance-Limited Regime} ($P \approx 41$), where generalization waits for the spectral gate to open; and (3) \textbf{Stability Override} ($P > 67$), where memorization becomes dimensionally unstable. Furthermore, we challenge the "Flat Minima" hypothesis for algorithmic tasks, showing that isotropic noise injection fails to induce grokking. Generalization requires the \textit{anisotropic rectification} unique to adaptive optimizers, which directs noise into the tangent space of the solution manifold.