Linear Spatial World Models Emerge in Large Language Models

TL;DR

This paper investigates whether large language models implicitly encode linear spatial world models, providing empirical evidence that such spatial representations emerge within their internal embeddings and are functionally utilized.

Contribution

The study introduces a formal framework for spatial world models and demonstrates their emergence in LLMs through probing and causal interventions.

Findings

LLMs encode linear spatial representations of object configurations.

Spatial representations are geometrically consistent and interpretable.

Causal interventions show these models are functionally used by LLMs.

Abstract

Large language models (LLMs) have demonstrated emergent abilities across diverse tasks, raising the question of whether they acquire internal world models. In this work, we investigate whether LLMs implicitly encode linear spatial world models, which we define as linear representations of physical space and object configurations. We introduce a formal framework for spatial world models and assess whether such structure emerges in contextual embeddings. Using a synthetic dataset of object positions, we train probes to decode object positions and evaluate geometric consistency of the underlying space. We further conduct causal interventions to test whether these spatial representations are functionally used by the model. Our results provide empirical evidence that LLMs encode linear spatial world models.