SceneCrafter: Controllable Multi-View Driving Scene Editing

TL;DR

SceneCrafter is a versatile multi-view editing framework that enables realistic, controllable, and 3D-consistent modifications of driving scenes from real data, enhancing simulation fidelity for autonomous vehicle development.

Contribution

The paper introduces SceneCrafter, a novel multi-view diffusion-based editing model that addresses key challenges in driving scene editing, including 3D consistency and multi-modality control, with new data generation techniques.

Findings

Achieves state-of-the-art realism and controllability in scene editing.

Demonstrates effective multi-modality manipulation including weather and time.

Ensures 3D consistency across multiple camera views.

Abstract

Simulation is crucial for developing and evaluating autonomous vehicle (AV) systems. Recent literature builds on a new generation of generative models to synthesize highly realistic images for full-stack simulation. However, purely synthetically generated scenes are not grounded in reality and have difficulty in inspiring confidence in the relevance of its outcomes. Editing models, on the other hand, leverage source scenes from real driving logs, and enable the simulation of different traffic layouts, behaviors, and operating conditions such as weather and time of day. While image editing is an established topic in computer vision, it presents fresh sets of challenges in driving simulation: (1) the need for cross-camera 3D consistency, (2) learning ``empty street" priors from driving data with foreground occlusions, and (3) obtaining paired image tuples of varied editing conditions while preserving consistent layout and geometry. To address these challenges, we propose SceneCrafter, a versatile editor for realistic 3D-consistent manipulation of driving scenes captured from multiple cameras. We build on recent advancements in multi-view diffusion models, using a fully controllable framework that scales seamlessly to multi-modality conditions like weather, time of day, agent boxes and high-definition maps. To generate paired data for supervising the editing model, we propose a novel framework on top of Prompt-to-Prompt to generate geometrically consistent synthetic paired data with global edits. We also introduce an alpha-blending framework to synthesize data with local edits, leveraging a model trained on empty street priors through novel masked training and multi-view repaint paradigm. SceneCrafter demonstrates powerful editing capabilities and achieves state-of-the-art realism, controllability, 3D consistency, and scene editing quality compared to existing baselines.