A Systematic Empirical Study of Grokking: Depth, Architecture, Activation, and Regularization

TL;DR

This paper systematically investigates the grokking phenomenon in neural networks, revealing that optimization stability and regularization, rather than architecture alone, primarily govern delayed generalization, with implications for model design.

Contribution

It provides a controlled, systematic empirical study disentangling architecture, optimization, and regularization effects on grokking, challenging prior architecture-centric views.

Findings

Depth has a non-monotonic effect on grokking.

Transformer and MLP differences largely vanish under matched hyperparameters.

Weight decay critically controls the grokking regime.

Abstract

Grokking the delayed transition from memorization to generalization in neural networks remains poorly understood, in part because prior empirical studies confound the roles of architecture, optimization, and regularization. We present a controlled study that systematically disentangles these factors on modular addition (mod 97), with matched and carefully tuned training regimes across models. Our central finding is that grokking dynamics are not primarily determined by architecture, but by interactions between optimization stability and regularization. Specifically, we show: (1) \textbf{depth has a non-monotonic effect}, with depth-4 MLPs consistently failing to grok while depth-8 residual networks recover generalization, demonstrating that depth requires architectural stabilization; (2) \textbf{the apparent gap between Transformers and MLPs largely disappears} (1.11$\times$ delay) under matched hyperparameters, indicating that previously reported differences are largely due to optimizer and regularization confounds; (3) \textbf{activation function effects are regime-dependent}, with GELU up to 4.3$\times$ faster than ReLU only when regularization permits memorization; and (4) \textbf{weight decay is the dominant control parameter}, exhibiting a narrow ``Goldilocks'' regime in which grokking occurs, while too little or too much prevents generalization. Across 3--5 seeds per configuration, these results provide a unified empirical account of grokking as an interaction-driven phenomenon. Our findings challenge architecture-centric interpretations and clarify how optimization and regularization jointly govern delayed generalization.