Grokking as Dimensional Phase Transition in Neural Networks

TL;DR

This paper models neural network grokking as a dimensional phase transition, revealing how effective dimensionality crossing relates to generalization onset and gradient field geometry.

Contribution

It introduces a finite-size scaling analysis of gradient avalanche dynamics, identifying grokking as a phase transition driven by gradient field geometry rather than architecture.

Findings

Grokking corresponds to a dimensional phase transition with effective dimensionality crossing from sub- to super-diffusive.

Gradient field geometry, not network architecture, determines the effective dimensionality D.

Synthetic Gaussian gradients maintain D ≈ 1, while real training shows excess D from backpropagation correlations.

Abstract

Neural network grokking -- the abrupt memorization-to-generalization transition -- challenges our understanding of learning dynamics. Through finite-size scaling of gradient avalanche dynamics across eight model scales, we find that grokking is a \textit{dimensional phase transition}: effective dimensionality~$D$ crosses from sub-diffusive (subcritical, $D < 1$) to super-diffusive (supercritical, $D > 1$) at generalization onset, exhibiting self-organized criticality (SOC). Crucially, $D$ reflects \textbf{gradient field geometry}, not network architecture: synthetic i.i.d.\ Gaussian gradients maintain $D \approx 1$ regardless of graph topology, while real training exhibits dimensional excess from backpropagation correlations. The grokking-localized $D(t)$ crossing -- robust across topologies -- offers new insight into the trainability of overparameterized networks.