Geometric Scaling of Bayesian Inference in LLMs

TL;DR

This paper demonstrates that large language models preserve a geometric structure enabling Bayesian inference, with last-layer representations correlating with predictive uncertainty across various models and prompts.

Contribution

It reveals that the geometric signature of Bayesian inference persists in production models and shows how this structure relates to uncertainty and model behavior.

Findings

Last-layer representations align along a dominant axis correlated with entropy.

Prompt restrictions collapse the geometric structure into low-dimensional manifolds.

Perturbing the entropy axis disrupts local uncertainty geometry without degrading Bayesian behavior.

Abstract

Recent work has shown that small transformers trained in controlled "wind-tunnel'' settings can implement exact Bayesian inference, and that their training dynamics produce a geometric substrate -- low-dimensional value manifolds and progressively orthogonal keys -- that encodes posterior structure. We investigate whether this geometric signature persists in production-grade language models. Across Pythia, Phi-2, Llama-3, and Mistral families, we find that last-layer value representations organize along a single dominant axis whose position strongly correlates with predictive entropy, and that domain-restricted prompts collapse this structure into the same low-dimensional manifolds observed in synthetic settings. To probe the role of this geometry, we perform targeted interventions on the entropy-aligned axis of Pythia-410M during in-context learning. Removing or perturbing this axis selectively disrupts the local uncertainty geometry, whereas matched random-axis interventions leave it intact. However, these single-layer manipulations do not produce proportionally specific degradation in Bayesian-like behavior, indicating that the geometry is a privileged readout of uncertainty rather than a singular computational bottleneck. Taken together, our results show that modern language models preserve the geometric substrate that enables Bayesian inference in wind tunnels, and organize their approximate Bayesian updates along this substrate.