SpaceMind: Camera-Guided Modality Fusion for Spatial Reasoning in Vision-Language Models

TL;DR

SpaceMind introduces a camera-guided modality fusion approach in vision-language models, significantly improving 3D spatial reasoning capabilities using only RGB inputs, and achieves state-of-the-art results on multiple benchmarks.

Contribution

The paper presents a novel camera-guided fusion module that enhances spatial reasoning in VLMs without relying on auxiliary 3D data, advancing multimodal understanding.

Findings

Achieves new state-of-the-art on VSI-Bench, SQA3D, and SPBench.

Outperforms existing methods on spatial reasoning tasks.

Demonstrates the effectiveness of camera-guided fusion for spatial grounding.

Abstract

Large vision-language models (VLMs) show strong multimodal understanding but still struggle with 3D spatial reasoning, such as distance estimation, size comparison, and cross-view consistency. Existing 3D-aware methods either depend on auxiliary 3D information or enhance RGB-only VLMs with geometry encoders through shallow feature fusion. We propose SpaceMind, a multimodal large language model explicitly designed for spatial reasoning solely from RGB inputs. The model adopts a dual-encoder architecture, integrating VGGT as a spatial understanding encoder and InternViT as a 2D visual encoder. The key idea is to treat the camera representation as an active guiding modality rather than passive metadata. Specifically, SpaceMind introduces a lightweight Camera-Guided Modality Fusion module before the language model to replace shallow fusion. It applies camera-conditioned biasing to spatial tokens, assigns query-independent weights reflecting their geometric importance, and uses the camera embedding to gate the fused representation. Empirically, SpaceMind establishes new state-of-the-art results on VSI-Bench, SQA3D and SPBench, surpassing both open and proprietary systems on VSI-Bench and SPBench by large margins and achieving state-of-the-art performance on SQA3D. These results demonstrate that camera-guided modality fusion is an effective and practical inductive bias for equipping VLMs with genuinely spatially grounded intelligence. We will release code and model checkpoints to support future research.