Implicit Statistical Inference in Transformers: Approximating Likelihood-Ratio Tests In-Context

TL;DR

This paper investigates how Transformers perform in-context learning by approximating likelihood-ratio tests, revealing they adaptively construct task-specific statistical estimators rather than relying on fixed heuristics.

Contribution

It introduces a statistical decision-theoretic framework to analyze Transformers, showing they approximate Bayes-optimal statistics and adapt their computations based on task complexity.

Findings

Transformers approximate Bayesian sufficient statistics in in-context learning.

Models adapt decision thresholds dynamically, not relying on fixed heuristics.

Performance matches ideal estimators in nonlinear regimes.

Abstract

In-context learning (ICL) allows Transformers to adapt to novel tasks without weight updates, yet the underlying algorithms remain poorly understood. We adopt a statistical decision-theoretic perspective by investigating simple binary hypothesis testing, where the optimal policy is determined by the likelihood-ratio test. Notably, this setup provides a mathematically rigorous setting for mechanistic interpretability where the target algorithmic ground truth is known. By training Transformers on tasks requiring distinct geometries (linear shifted means vs. nonlinear variance estimation), we demonstrate that the models approximate the Bayes-optimal sufficient statistics from context up to some monotonic transformation, matching the performance of an ideal oracle estimator in nonlinear regimes. Leveraging this analytical ground truth, mechanistic analysis via logit lens and circuit alignment suggests that the model does not rely on a fixed kernel smoothing heuristic. Instead, it appears to adapt the point at which decisions become linearly decodable: exhibiting patterns consistent with a voting-style ensemble for linear tasks while utilizing a deeper sequential computation for nonlinear tasks. These findings suggest that ICL emerges from the construction of task-adaptive statistical estimators rather than simple similarity matching.