Depth-Wise Emergence of Prediction-Centric Geometry in Large Language Models

TL;DR

This paper reveals how large language models transition from processing context to predicting tokens through a depth-wise geometric reorganization of their internal representations, enabling causal control over predictions.

Contribution

It introduces a unified geometric framework to analyze the depth-wise transition in LLMs and demonstrates how late-layer representations facilitate prediction control.

Findings

Representation geometry parametrizes prediction similarity.

Representation norms encode context-specific information.

A mechanistic account of context-to-prediction transformation.

Abstract

We show that decoder-only large language models exhibit a depth-wise transition from context-processing to prediction-forming phases of computation accompanied by a reorganization of representational geometry. Using a unified framework combining geometric analysis with mechanistic intervention, we demonstrate that late-layer representations implement a structured geometric code that enables selective causal control over token prediction. Specifically, angular organization of the representation geometry parametrizes prediction distributional similarity, while representation norms encode context-specific information that does not determine prediction. Together, these results provide a mechanistic-geometric account of the dynamics of transforming context into predictions in LLMs.