Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learning

TL;DR

This paper investigates the mechanisms behind in-context learning in large language models, revealing it is neither just memorization nor a symbolic algorithm, but a complex interplay of factors clarified through systematic analysis.

Contribution

The study introduces a new suite of investigative tasks and a novel method to analyze in-context learning, providing deeper mechanistic insights beyond prior assumptions.

Findings

ICL extends beyond simple memorization of training data.

ICL does not implement an independent symbolic algorithm.

Training dynamics and model capabilities influence ICL performance.

Abstract

Large-scale Transformer language models (LMs) trained solely on next-token prediction with web-scale data can solve a wide range of tasks after seeing just a few examples. The mechanism behind this capability, known as in-context learning (ICL), remains both controversial and poorly understood. Some studies argue that it is merely the result of memorizing vast amounts of data, while others contend that it reflects a fundamental, symbolic algorithmic development in LMs. In this work, we introduce a suite of investigative tasks and a novel method to systematically investigate ICL by leveraging the full Pythia scaling suite, including interim checkpoints that capture progressively larger amount of training data. By carefully exploring ICL performance on downstream tasks and simultaneously conducting a mechanistic analysis of the residual stream's subspace, we demonstrate that ICL extends beyond mere "memorization" of the training corpus, yet does not amount to the implementation of an independent symbolic algorithm. Our results also clarify several aspects of ICL, including the influence of training dynamics, model capabilities, and elements of mechanistic interpretability. Overall, our work advances the understanding of ICL and its implications, offering model developers insights into potential improvements and providing AI security practitioners with a basis for more informed guidelines.