Uncovering and Shaping the Latent Representation of 3D Scene Topology in Vision-Language Models

TL;DR

This paper reveals that vision-language models contain a latent 3D topological map of scenes, which can be mathematically shaped and regularized to improve spatial reasoning tasks.

Contribution

It uncovers the geometric nature of VLMs' latent representations and introduces a regularization method that enhances spatial understanding.

Findings

Latent topological maps in VLMs are overshadowed by visual semantics.

A linear feature extraction isolates a pure spatial subspace.

Regularization improves spatial task performance by up to 12.1%.

Abstract

Decades of cognitive science establish that humans navigate environments by forming cognitive maps, defined as allocentric and topology-preserving representations of 3D space. While modern Vision-Language Models (VLMs) demonstrate emergent spatial reasoning from 2D egocentric inputs, it remains unclear whether they construct an analogous 3D internal representation. In this paper, we demonstrate that current VLMs do possess a latent topological map of 3D scenes, but it is heavily overshadowed by non-geometric visual semantics, such as color and shape. By isolating this spatial subspace through cross-scene linear feature extraction, we extract a clean spatial subspace that causally controls the model's spatial outputs. We mathematically shape this latent representation and prove its correspondence to the Laplacian eigenmaps of the scene's 3D Gaussian-kernel graph, converging to the physical 3D space in the continuous limit. Motivated by this geometric identification, we further introduce a mathematically principled latent regularization method for VLMs, based on Dirichlet energy. Applying this single-term regularizer to a minimal 500-step supervised VLM fine-tuning (SFT) on simple synthetic data yields significant improvements on real-world spatial benchmarks, outperforming standard SFT and competitive baselines by up to 12.1\% in spatial tasks involving scene topology understanding. Source code is available at https://github.com/pittisl/vlm-latent-shaping