GeoWorld-VLM: Geometry from World Models for Vision-Language Models

TL;DR

GeoWorld-VLM enhances vision-language models' spatial reasoning by distilling 3D geometric cues from world models into the visual pathway, improving spatial understanding without altering language capabilities.

Contribution

Introduces a novel distillation framework that transfers geometric information from world models into VLMs, improving spatial reasoning across different architectures and datasets.

Findings

Improves spatial reasoning performance by ~4% on benchmarks.

Enhances visual pathway with geometric cues without affecting language understanding.

Demonstrates generality across multiple VLM architectures.

Abstract

Modern Vision-Language Models (VLMs) achieve strong semantic recognition, yet remain brittle on elementary spatial relations such as left of, on, behind, and between. One cause of this failure arises before language reasoning begins: the visual pathway may compress or discard critical 3D structural cues during feature extraction, so the language model receives image representations that are already insufficient for reliable spatial judgment. We introduce GeoWorld-VLM, a VLM-side distillation framework that transfers geometric structure from frozen camera-conditioned video world models into VLMs. GeoWorld-VLM fine-tunes only the image encoder and multimodal projector, aligning post-projector image features with intermediate world-model representations while leaving the main backbone frozen. Given images, a prompt, and a sampled camera trajectory, the world-model teacher converts static visual input into a synthetic multi-view spatial signal. Training combines spatial answer supervision, teacher-student feature alignment, and a preservation anchor to the original VLM. Since the language model remains frozen, GeoWorld-VLM preserves the original model's linguistic capabilities while attributing spatial improvements to the enhanced visual pathway. To evaluate the effectiveness and generality of the proposed method, we apply GeoWorld-VLM to two distinct VLM architectures and observe consistent improvements across both backbones. GeoWorld-VLM improves performance by approximately 4 percent on both the What'sUp and VSR benchmarks, suggesting that world-model-guided visual alignment generalizes across model structures and spatial reasoning datasets.