Spectral Edge Dynamics Reveal Functional Modes of Learning

TL;DR

This paper shows that training dynamics in neural networks concentrate along a small set of spectral directions that reveal functional modes of learning, which are not captured by traditional interpretability tools.

Contribution

It introduces spectral edge analysis to uncover low-dimensional functional modes of learning, revealing how task symmetry influences these modes.

Findings

Spectral edges reliably distinguish grokking from non-grokking regimes.

Functional modes vary with task symmetry, with simple harmonic structures emerging in symmetric tasks.

Multitask training enhances the concentration of spectral modes, reflecting compositional structure.

Abstract

Training dynamics during grokking concentrate along a small number of dominant update directions -- the spectral edge -- which reliably distinguishes grokking from non-grokking regimes. We show that standard mechanistic interpretability tools (head attribution, activation probing, sparse autoencoders) fail to capture these directions: their structure is not localized in parameter or feature space. Instead, each direction induces a structured function over the input domain, revealing low-dimensional functional modes invisible to representation-level analysis. For modular addition, all leading directions collapse to a single Fourier mode. For multiplication, the same collapse appears only in the discrete-log basis, yielding a 5.9x improvement in concentration. For subtraction, the edge spans a small multi-mode family. For $x^2+y^2$, no single harmonic basis suffices, but cross-terms of additive and multiplicative features provide a 4x variance boost, consistent with the decomposition (a+b)^2 - 2ab. Multitask training amplifies this compositional structure, with the $x^2+y^2$ spectral edge inheriting the addition circuit's characteristic frequency (2.3x concentration increase). These results suggest that training discovers low-dimensional functional modes over the input domain, whose structure depends on the algebraic symmetry of the task. These results suggest that spectral edge dynamics identify low-dimensional functional subspaces governing learning, whose representation depends on the algebraic structure of the task. Simple harmonic structure emerges only when the task admits a symmetry-adapted basis; more complex tasks require richer functional descriptions.