Mirage2Matter: A Physically Grounded Gaussian World Model from Video

TL;DR

This paper introduces a scalable, photorealistic world modeling framework using multi-view videos and Gaussian Splatting, enabling effective simulation for embodied AI without expensive sensors.

Contribution

It presents a novel simulation framework that reconstructs real environments into high-fidelity, physically grounded models from videos, improving scalability and practicality for embodied AI training.

Findings

Vision Language Action models trained on simulated data perform well zero-shot.

The framework achieves high-fidelity scene reconstruction from videos.

Simulated data matches or surpasses real-world data in downstream tasks.

Abstract

The scalability of embodied intelligence is fundamentally constrained by the scarcity of real-world interaction data. While simulation platforms provide a promising alternative, existing approaches often suffer from a substantial visual and physical gap to real environments and rely on expensive sensors, precise robot calibration, or depth measurements, limiting their practicality at scale. We present Simulate Anything, a graphics-driven world modeling and simulation framework that enables efficient generation of high-fidelity embodied training data using only multi-view environment videos and off-the-shelf assets. Our approach reconstructs real-world environments into a photorealistic scene representation using 3D Gaussian Splatting (3DGS), seamlessly capturing fine-grained geometry and appearance from video. We then leverage generative models to recover a physically realistic representation and integrate it into a simulation environment via a precision calibration target, enabling accurate scale alignment between the reconstructed scene and the real world. Together, these components provide a unified, editable, and physically grounded world model. Vision Language Action (VLA) models trained on our simulated data achieve strong zero-shot performance on downstream tasks, matching or even surpassing results obtained with real-world data, highlighting the potential of reconstruction-driven world modeling for scalable and practical embodied intelligence training.