Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models

TL;DR

This paper uncovers a linear mechanism in vision-language models that encodes spatial and temporal information through IDs, enabling better interpretability and understanding of their reasoning processes.

Contribution

It identifies and characterizes a linear spatiotemporal ID mechanism in VLMs, advancing interpretability and causal understanding of their reasoning capabilities.

Findings

VLMs encode object locations via linear spatial IDs

Spatial IDs mediate model beliefs at intermediate layers

Analogous temporal IDs found in video VLMs

Abstract

Spatio-temporal reasoning is a remarkable capability of Vision Language Models (VLMs), but the underlying mechanisms of such abilities remain largely opaque. We postulate that visual/geometrical and textual representations of spatial structure must be combined at some point in VLM computations. We search for such confluence, and ask whether the identified representation can causally explain aspects of input-output model behavior through a linear model. We show empirically that VLMs encode object locations by linearly binding \textit{spatial IDs} to textual activations, then perform reasoning via language tokens. Through rigorous causal interventions we demonstrate that these IDs, which are ubiquitous across the model, can systematically mediate model beliefs at intermediate VLM layers. Additionally, we find that spatial IDs serve as a diagnostic tool for identifying limitations in existing VLMs, and as a valuable learning signal. We extend our analysis to video VLMs and identify an analogous linear temporal ID mechanism. By characterizing our proposed spatiotemporal ID mechanism, we elucidate a previously underexplored internal reasoning process in VLMs, toward improved interpretability and the principled design of more aligned and capable models. We release our code for reproducibility: https://github.com/Raphoo/linear-mech-vlms.