Physical Informed Driving World Model

TL;DR

DrivePhysica is a physics-informed generative model for multi-view driving videos that enhances realism and consistency by integrating physical principles into the generation process, improving downstream perception tasks.

Contribution

The paper introduces DrivePhysica, a novel model that incorporates physical principles into driving video generation through three modules, advancing realism and spatial-temporal consistency.

Findings

Achieves state-of-the-art FID and FVD scores on Nuscenes dataset.

Improves downstream perception task performance.

Ensures realistic motion and occlusion handling in generated videos.

Abstract

Autonomous driving requires robust perception models trained on high-quality, large-scale multi-view driving videos for tasks like 3D object detection, segmentation and trajectory prediction. While world models provide a cost-effective solution for generating realistic driving videos, challenges remain in ensuring these videos adhere to fundamental physical principles, such as relative and absolute motion, spatial relationship like occlusion and spatial consistency, and temporal consistency. To address these, we propose DrivePhysica, an innovative model designed to generate realistic multi-view driving videos that accurately adhere to essential physical principles through three key advancements: (1) a Coordinate System Aligner module that integrates relative and absolute motion features to enhance motion interpretation, (2) an Instance Flow Guidance module that ensures precise temporal consistency via efficient 3D flow extraction, and (3) a Box Coordinate Guidance module that improves spatial relationship understanding and accurately resolves occlusion hierarchies. Grounded in physical principles, we achieve state-of-the-art performance in driving video generation quality (3.96 FID and 38.06 FVD on the Nuscenes dataset) and downstream perception tasks. Our project homepage: https://metadrivescape.github.io/papers_project/DrivePhysica/page.html