Provable Low-Frequency Bias of In-Context Learning of Representations

TL;DR

This paper provides a theoretical explanation for in-context learning in large language models, showing it biases representations towards low-frequency, smooth structures, and demonstrates robustness to high-frequency noise.

Contribution

It introduces a unified double convergence framework explaining ICL's low-frequency bias and validates this with analytical proofs and empirical experiments.

Findings

Representations converge over context and layers, leading to low-frequency bias.

ICL representations are robust against high-frequency noise.

Total energy of representations decays without vanishing.

Abstract

In-context learning (ICL) enables large language models (LLMs) to acquire new behaviors from the input sequence alone without any parameter updates. Recent studies have shown that ICL can surpass the original meaning learned in pretraining stage through internalizing the structure the data-generating process (DGP) of the prompt into the hidden representations. However, the mechanisms by which LLMs achieve this ability is left open. In this paper, we present the first rigorous explanation of such phenomena by introducing a unified framework of double convergence, where hidden representations converge both over context and across layers. This double convergence process leads to an implicit bias towards smooth (low-frequency) representations, which we prove analytically and verify empirically. Our theory explains several open empirical observations, including why learned representations exhibit globally structured but locally distorted geometry, and why their total energy decays without vanishing. Moreover, our theory predicts that ICL has an intrinsic robustness towards high-frequency noise, which we empirically confirm. These results provide new insights into the underlying mechanisms of ICL, and a theoretical foundation to study it that hopefully extends to more general data distributions and settings.