Model Capacity Determines Grokking through Competing Memorisation and Generalisation Speeds

TL;DR

This paper presents an information-theoretic framework explaining grokking as a competition between memorisation and generalisation speeds, both dependent on model capacity, on modular arithmetic tasks.

Contribution

It introduces a formal model linking model size to grokking through measurable timescales, advancing understanding of how capacity influences learning dynamics.

Findings

Grokking occurs near the intersection of memorisation and generalisation timescales.

Larger models memorise faster, consistent with empirical observations.

The framework predicts memorisation speed based on model capacity and dataset complexity.

Abstract

Existing accounts of grokking explain the phenomena in terms of mechanistic frameworks such as circuit efficiency or lazy-to-rich transitions. However, despite a known dependence between grokking and model size, how model capacity shapes grokking remains an open question. We give an information-theoretic account of this relationship on the task of modular arithmetic, showing that grokking does not immediately occur when a model becomes large enough to memorise the training set, but rather emerges as the outcome of a competition between two measurable timescales: a memorisation speed $T_{\text{mem}}(P)$ and a generalisation speed $T_{\text{gen}}(P)$, both of which are functions of model parameter count $P$. Adapting the information capacity framework of Morris et al. (2025), we estimate $T_{\text{mem}}(P)$ on random-label data of equivalent complexity and $T_{\text{gen}}(P)$ on the modular task itself, and show that grokking emerges close to the parameter scale where these timescales intersect. The framework also suggests an empirical model for predicting memorisation speed given model capacity and dataset complexity, recovering the previously reported empirical observation that larger models memorise faster. Overall, we motivate the formalisation of different learning timescales as important abstractions to study when explaining how model capacity shapes grokking on algorithmic tasks.