Emergent Structured Representations Support Flexible In-Context Inference in Large Language Models

TL;DR

This paper uncovers how large language models internally develop and utilize structured latent representations in middle to late layers to perform flexible in-context inference across diverse tasks.

Contribution

It demonstrates that LLMs dynamically construct a causal, structured conceptual subspace that is central to their reasoning process during inference.

Findings

A conceptual subspace emerges in middle to late layers of LLMs.

Causal mediation shows this subspace is essential for model predictions.

Attention heads in early layers build and refine the subspace for inference.

Abstract

Large language models (LLMs) exhibit emergent behaviors suggestive of human-like reasoning. While recent work has identified structured conceptual representations within these models, it remains unclear whether they functionally rely on such representations for reasoning. Here we investigate the internal processing of LLMs during in-context inference across diverse tasks. Our results reveal a conceptual subspace emerging in middle to late layers, whose representational structure persists across contexts. Using causal mediation analyses, we demonstrate that this subspace is not merely an epiphenomenon but is functionally central to model predictions, establishing its causal role in inference. We further identify a layer-wise progression where attention heads in early-to-middle layers integrate contextual cues to construct and refine the subspace, which is subsequently leveraged by later layers to generate predictions. Together, these findings provide evidence that LLMs dynamically construct and use structured latent representations in context for inference, offering insights into the computational processes underlying flexible adaptation.