Grokking Explained: A Statistical Phenomenon

TL;DR

This paper investigates the grokking phenomenon in deep learning, revealing that distribution shifts between training and test data are key, and introduces synthetic datasets to analyze its causes and mechanisms.

Contribution

It formalizes grokking, demonstrates its relation to distribution shifts, and shows it can occur with dense data and minimal tuning, advancing understanding of this phenomenon.

Findings

Grokking is linked to distribution shifts between training and test data.

Small-sampling facilitates grokking but is not its primary cause.

Grokking can occur with dense data and minimal hyper-parameter tuning.

Abstract

Grokking, or delayed generalization, is an intriguing learning phenomenon where test set loss decreases sharply only after a model's training set loss has converged. This challenges conventional understanding of the training dynamics in deep learning networks. In this paper, we formalize and investigate grokking, highlighting that a key factor in its emergence is a distribution shift between training and test data. We introduce two synthetic datasets specifically designed to analyze grokking. One dataset examines the impact of limited sampling, and the other investigates transfer learning's role in grokking. By inducing distribution shifts through controlled imbalanced sampling of sub-categories, we systematically reproduce the phenomenon, demonstrating that while small-sampling is strongly associated with grokking, it is not its cause. Instead, small-sampling serves as a convenient mechanism for achieving the necessary distribution shift. We also show that when classes form an equivariant map, grokking can be explained by the model's ability to learn from similar classes or sub-categories. Unlike earlier work suggesting that grokking primarily arises from high regularization and sparse data, we demonstrate that it can also occur with dense data and minimal hyper-parameter tuning. Our findings deepen the understanding of grokking and pave the way for developing better stopping criteria in future training processes.