MLLM-4D: Towards Visual-based Spatial-Temporal Intelligence

TL;DR

MLLM-4D introduces a novel framework that enhances multimodal large language models with 4D spatial-temporal reasoning from visual inputs, utilizing new data curation and training strategies.

Contribution

The paper presents a cost-effective data curation pipeline and a post-training strategy that significantly improve 4D understanding and reasoning in MLLMs without architectural modifications.

Findings

Achieves state-of-the-art spatial-temporal reasoning from 2D RGB inputs.

Develops high-quality 4D spatiotemporal datasets from stereo videos.

Demonstrates effective 4D reasoning capabilities through extensive experiments.

Abstract

Humans are born with vision-based 4D spatial-temporal intelligence, which enables us to perceive and reason about the evolution of 3D space over time from purely visual inputs. Despite its importance, this capability remains a significant bottleneck for current multimodal large language models (MLLMs). To tackle this challenge, we introduce MLLM-4D, a comprehensive framework designed to bridge the gaps in training data curation and model post-training for spatiotemporal understanding and reasoning. On the data front, we develop a cost-efficient data curation pipeline that repurposes existing stereo video datasets into high-quality 4D spatiotemporal instructional data. This results in the MLLM4D-2M and MLLM4D-R1-30k datasets for Supervised Fine-Tuning (SFT) and Reinforcement Fine-Tuning (RFT), alongside MLLM4D-Bench for comprehensive evaluation. Regarding model training, our post-training strategy establishes a foundational 4D understanding via SFT and further catalyzes 4D reasoning capabilities by employing Group Relative Policy Optimization (GRPO) with specialized Spatiotemporal Chain of Thought (ST-CoT) prompting and Spatiotemporal reward functions (ST-reward) without involving the modification of architecture. Extensive experiments demonstrate that MLLM-4D achieves state-of-the-art spatial-temporal understanding and reasoning capabilities from purely 2D RGB inputs. Project page: https://github.com/GVCLab/MLLM-4D.