Martian World Model: Controllable Video Synthesis with Physically Accurate 3D Reconstructions

TL;DR

This paper introduces a comprehensive system for synthesizing realistic Martian landscape videos by reconstructing 3D environments from NASA data and generating consistent videos conditioned on initial frames and prompts.

Contribution

It presents a novel pipeline combining 3D reconstruction from Martian data with a controllable video generator, addressing data scarcity and domain gap issues.

Findings

Outperforms terrestrial-trained models in visual fidelity

Produces physically accurate 3D surface models of Mars

Enables controllable video synthesis with prompts and trajectories

Abstract

Synthesizing realistic Martian landscape videos is crucial for mission rehearsal and robotic simulation. However, this task poses unique challenges due to the scarcity of high-quality Martian data and the significant domain gap between Martian and terrestrial imagery. To address these challenges, we propose a holistic solution composed of two key components: 1) A data curation pipeline Multimodal Mars Synthesis (M3arsSynth), which reconstructs 3D Martian environments from real stereo navigation images, sourced from NASA's Planetary Data System (PDS), and renders high-fidelity multiview 3D video sequences. 2) A Martian terrain video generator, MarsGen, which synthesizes novel videos visually realistic and geometrically consistent with the 3D structure encoded in the data. Our M3arsSynth engine spans a wide range of Martian terrains and acquisition dates, enabling the generation of physically accurate 3D surface models at metric-scale resolution. MarsGen, fine-tuned on M3arsSynth data, synthesizes videos conditioned on an initial image frame and, optionally, camera trajectories or textual prompts, allowing for video generation in novel environments. Experimental results show that our approach outperforms video synthesis models trained on terrestrial datasets, achieving superior visual fidelity and 3D structural consistency.