Critical Data Size of Language Models from a Grokking Perspective

TL;DR

This paper investigates the critical data size needed for language models to transition from memorization to generalization, revealing a size-dependent phase transition and proposing a formal data efficiency hypothesis.

Contribution

It formalizes the phase transition in language model training, introduces the Data Efficiency Hypothesis, and demonstrates how model size influences the critical data threshold for generalization.

Findings

Generalization occurs only after reaching a critical data size.

Larger models require more data to reach the critical size.

Smoother phase transitions are observed at the critical dataset size.

Abstract

We explore the critical data size in language models, a threshold that marks a fundamental shift from quick memorization to slow generalization. We formalize the phase transition under the grokking configuration into the Data Efficiency Hypothesis and identify data insufficiency, sufficiency, and surplus regimes in language models training dynamics. We develop a grokking configuration to reproduce grokking on simplistic language models stably by rescaling initialization and weight decay. We show that generalization occurs only when language models reach a critical size. We analyze grokking across sample-wise and model-wise, verifying the proposed data efficiency hypothesis. Our experiments reveal smoother phase transitions occurring at the critical dataset size for language datasets. As the model size increases, this critical point also becomes larger, indicating that larger models require more data. Our results deepen the understanding of language model training, offering a novel perspective on the role of data in the learning mechanism of language models.