Predicting the Emergence of Induction Heads in Language Model Pretraining

TL;DR

This paper investigates how induction heads emerge in language models, revealing that their formation depends on training data statistics like bigram repetition and context size, regardless of model size.

Contribution

It provides a predictive equation for induction head emergence and analyzes how data properties influence their formation in language models.

Findings

Emergence point predicted by batch size and context size equation

Bigram repetition frequency and reliability strongly influence IH formation

Local dependency with high bigram repetition suffices for IH emergence

Abstract

Specialized attention heads dubbed induction heads (IHs) have been argued to underlie the remarkable in-context learning capabilities of modern language models; yet, a precise characterization of their emergence, especially in the context of language modeling, remains wanting. In this study, we investigate the relationship between statistical properties of the training data and IH formation in both natural and synthetic training data settings. We show that: (1) A simple equation combining batch size and context size predicts the point at which IHs form and that this emergence point is agnostic to model size; (2) Surface bigram repetition frequency and reliability strongly affect the formation of IHs, and we find an effective Pareto frontier in terms of these two values; (3) local dependency with high bigram repetition frequency and reliability is sufficient for IH formation, but when the frequency and reliability are low, categoriality and the shape of the marginal distribution matter.