Stag-1: Towards Realistic 4D Driving Simulation with Video Generation Model

TL;DR

Stag-1 is a novel 4D driving simulation model that reconstructs real-world scenes and generates photo-realistic, controllable videos from any viewpoint, advancing autonomous driving simulation realism.

Contribution

The paper introduces Stag-1, a new model that reconstructs continuous 4D scenes and produces controllable, photo-realistic videos, addressing view transformation and dynamic modeling challenges.

Findings

Effective multi-view scene consistency

High background coherence in generated videos

Accurate scene evolution modeling

Abstract

4D driving simulation is essential for developing realistic autonomous driving simulators. Despite advancements in existing methods for generating driving scenes, significant challenges remain in view transformation and spatial-temporal dynamic modeling. To address these limitations, we propose a Spatial-Temporal simulAtion for drivinG (Stag-1) model to reconstruct real-world scenes and design a controllable generative network to achieve 4D simulation. Stag-1 constructs continuous 4D point cloud scenes using surround-view data from autonomous vehicles. It decouples spatial-temporal relationships and produces coherent keyframe videos. Additionally, Stag-1 leverages video generation models to obtain photo-realistic and controllable 4D driving simulation videos from any perspective. To expand the range of view generation, we train vehicle motion videos based on decomposed camera poses, enhancing modeling capabilities for distant scenes. Furthermore, we reconstruct vehicle camera trajectories to integrate 3D points across consecutive views, enabling comprehensive scene understanding along the temporal dimension. Following extensive multi-level scene training, Stag-1 can simulate from any desired viewpoint and achieve a deep understanding of scene evolution under static spatial-temporal conditions. Compared to existing methods, our approach shows promising performance in multi-view scene consistency, background coherence, and accuracy, and contributes to the ongoing advancements in realistic autonomous driving simulation. Code: https://github.com/wzzheng/Stag.