Scaling Laws and In-Context Learning: A Unified Theoretical Framework

TL;DR

This paper presents a unified theoretical framework linking scaling laws to in-context learning in transformers, revealing how model size and structure influence learning capabilities and phase transitions.

Contribution

It introduces a comprehensive theory connecting scaling laws to ICL emergence, including phase transitions and optimal model size allocations, validated by systematic experiments.

Findings

ICL performance follows power-law scaling with model parameters.

Transformers implement gradient-based meta-learning in their forward pass.

Sharp phase transitions occur at critical model scales.

Abstract

In-context learning (ICL) enables large language models to adapt to new tasks from demonstrations without parameter updates. Despite extensive empirical studies, a principled understanding of ICL emergence at scale remains more elusive. We present a unified theoretical framework connecting scaling laws to ICL emergence in transformers. Our analysis establishes that ICL performance follows power-law relationships with model depth $L$, width $d$, context length $k$, and training data $D$, with exponents determined by task structure. We show that under specific conditions, transformers implement gradient-based metalearning in their forward pass, with an effective learning rate $\eta_{\text{eff}} = \Theta(1/\sqrt{Ld})$. We demonstrate sharp phase transitions at critical scales and derive optimal depth-width allocations favoring $L^* \propto N^{2/3}$, $d^* \propto N^{1/3}$ for the fixed parameter budget $N = Ld$. Systematic experiments on synthetic tasks validate our predictions, with measured scaling exponents closely matching theory. This work provides both necessary and sufficient conditions for the emergence of ICLs and establishes fundamental computational limits on what transformers can learn in-context.