The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains

TL;DR

This paper investigates how large language models develop in-context learning abilities for Markov chains, revealing a multi-phase training process and the emergence of statistical induction heads that predict based on bigram statistics.

Contribution

It introduces a Markov Chain sequence modeling task to analyze in-context learning, providing empirical and theoretical insights into the phases of model training and the emergence of statistical induction heads.

Findings

Models pass through multiple training phases from uniform to bigram prediction.

Presence of unigram solutions may delay bigram solution formation.

Learning dynamics are influenced by the prior distribution over Markov chains.

Abstract

Large language models have the ability to generate text that mimics patterns in their inputs. We introduce a simple Markov Chain sequence modeling task in order to study how this in-context learning (ICL) capability emerges. In our setting, each example is sampled from a Markov chain drawn from a prior distribution over Markov chains. Transformers trained on this task form \emph{statistical induction heads} which compute accurate next-token probabilities given the bigram statistics of the context. During the course of training, models pass through multiple phases: after an initial stage in which predictions are uniform, they learn to sub-optimally predict using in-context single-token statistics (unigrams); then, there is a rapid phase transition to the correct in-context bigram solution. We conduct an empirical and theoretical investigation of this multi-phase process, showing how successful learning results from the interaction between the transformer's layers, and uncovering evidence that the presence of the simpler unigram solution may delay formation of the final bigram solution. We examine how learning is affected by varying the prior distribution over Markov chains, and consider the generalization of our in-context learning of Markov chains (ICL-MC) task to $n$-grams for $n > 2$.