Latent Algorithmic Structure Precedes Grokking: A Mechanistic Study of ReLU MLPs on Modular Arithmetic

TL;DR

This paper investigates the internal mechanisms of ReLU MLPs during grokking, revealing that they develop near-binary square wave input weights and phase-aligned output weights, which are key to their eventual generalization.

Contribution

It uncovers the latent algorithmic structure in ReLU MLPs that precedes grokking, showing how weights evolve into binary and phase-aligned forms, and introduces an idealized model capturing this behavior.

Findings

ReLU MLPs learn near-binary square wave input weights.

Output weights exhibit phase-sum relations satisfying  + .

Idealized model achieves high accuracy using extracted Fourier components.

Abstract

Grokking-the phenomenon where validation accuracy of neural networks on modular addition of two integers rises long after training data has been memorized-has been characterized in previous works as producing sinusoidal input weight distributions in transformers and multi-layer perceptrons (MLPs). We find empirically that ReLU MLPs in our experimental setting instead learn near-binary square wave input weights, where intermediate-valued weights appear exclusively near sign-change boundaries, alongside output weight distributions whose dominant Fourier phases satisfy a phase-sum relation $\phi_{\mathrm{out}} = \phi_a + \phi_b$; this relation holds even when the model is trained on noisy data and fails to grok. We extract the frequency and phase of each neuron's weights via DFT and construct an idealized MLP: Input weights are replaced by perfect binary square waves and output weights by cosines, both parametrized by the frequencies, phases, and amplitudes extracted from the dominant Fourier components of the real model weights. This idealized model achieves 95.5% accuracy when the frequencies and phases are extracted from the weights of a model trained on noisy data that itself achieves only 0.23% accuracy. This suggests that grokking does not discover the correct algorithm, but rather sharpens an algorithm substantially encoded during memorization, progressively binarizing the input weights into cleaner square waves and aligning the output weights, until generalization becomes possible.