Learning to Reason in 4D: Dynamic Spatial Understanding for Vision Language Models

TL;DR

This paper introduces DSR Suite, a comprehensive framework including a new dataset, benchmark, and model enhancements for improving vision-language models' ability to perform dynamic spatial reasoning in 4D from in-the-wild videos.

Contribution

The paper presents a scalable pipeline for generating 4D-aware training data, a new benchmark for dynamic spatial reasoning, and a lightweight module to incorporate geometric priors into vision-language models.

Findings

Enhanced dynamic spatial reasoning in VLMs with DSR-Train and GSM.

Maintained performance on general video understanding tasks.

Significant improvement in 4D spatial reasoning capabilities.

Abstract

Vision-language models (VLM) excel at general understanding yet remain weak at dynamic spatial reasoning (DSR), i.e., reasoning about the evolvement of object geometry and relationship in 3D space over time, largely due to the scarcity of scalable 4D-aware training resources. To bridge this gap across aspects of dataset, benchmark and model, we introduce DSR Suite. First, we propose an automated pipeline that generates multiple-choice question-answer pairs from in-the-wild videos for DSR. By leveraging modern vision foundation models, the pipeline extracts rich geometric and motion information, including camera poses, local point clouds, object masks, orientations, and 3D trajectories. These geometric cues enable the construction of DSR-Train for learning and further human-refined DSR-Bench for evaluation. Compared with previous works, our data emphasize (i) in-the-wild video sources, (ii) object- and scene-level 3D requirements, (iii) viewpoint transformations, (iv) multi-object interactions, and (v) fine-grained, procedural answers. Beyond data, we propose a lightweight Geometry Selection Module (GSM) to seamlessly integrate geometric priors into VLMs, which condenses question semantics and extracts question-relevant knowledge from pretrained 4D reconstruction priors into a compact set of geometry tokens. This targeted extraction avoids overwhelming the model with irrelevant knowledge. Experiments show that integrating DSR-Train and GSM into Qwen2.5-VL-7B significantly enhances its dynamic spatial reasoning capability, while maintaining accuracy on general video understanding benchmarks.