What learning algorithm is in-context learning? Investigations with linear models

TL;DR

This paper investigates whether transformer-based in-context learners implicitly implement standard linear learning algorithms, providing theoretical proofs and empirical evidence that they approximate gradient descent, ridge regression, and Bayesian estimators.

Contribution

It demonstrates that transformers can encode and execute classical linear learning algorithms internally, bridging neural in-context learning with traditional statistical methods.

Findings

Transformers can implement linear learning algorithms like gradient descent and ridge regression.

Trained in-context learners closely match predictors from classical algorithms.

Learners' internal representations encode weights and matrices similar to those in traditional estimators.

Abstract

Neural sequence models, especially transformers, exhibit a remarkable capacity for in-context learning. They can construct new predictors from sequences of labeled examples $(x, f(x))$ presented in the input without further parameter updates. We investigate the hypothesis that transformer-based in-context learners implement standard learning algorithms implicitly, by encoding smaller models in their activations, and updating these implicit models as new examples appear in the context. Using linear regression as a prototypical problem, we offer three sources of evidence for this hypothesis. First, we prove by construction that transformers can implement learning algorithms for linear models based on gradient descent and closed-form ridge regression. Second, we show that trained in-context learners closely match the predictors computed by gradient descent, ridge regression, and exact least-squares regression, transitioning between different predictors as transformer depth and dataset noise vary, and converging to Bayesian estimators for large widths and depths. Third, we present preliminary evidence that in-context learners share algorithmic features with these predictors: learners' late layers non-linearly encode weight vectors and moment matrices. These results suggest that in-context learning is understandable in algorithmic terms, and that (at least in the linear case) learners may rediscover standard estimation algorithms. Code and reference implementations are released at https://github.com/ekinakyurek/google-research/blob/master/incontext.