DriveEditor: A Unified 3D Information-Guided Framework for Controllable Object Editing in Driving Scenes

TL;DR

DriveEditor is a diffusion-based framework that enables precise and diverse object editing in driving videos, including repositioning, replacement, deletion, and insertion, while maintaining high fidelity and consistency.

Contribution

It introduces a unified, diffusion-based approach with novel position control and appearance maintenance modules tailored for driving scene editing.

Findings

Achieves high-fidelity object manipulation in driving videos

Demonstrates superior controllability and diversity in scene editing

Facilitates downstream tasks with realistic scene modifications

Abstract

Vision-centric autonomous driving systems require diverse data for robust training and evaluation, which can be augmented by manipulating object positions and appearances within existing scene captures. While recent advancements in diffusion models have shown promise in video editing, their application to object manipulation in driving scenarios remains challenging due to imprecise positional control and difficulties in preserving high-fidelity object appearances. To address these challenges in position and appearance control, we introduce DriveEditor, a diffusion-based framework for object editing in driving videos. DriveEditor offers a unified framework for comprehensive object editing operations, including repositioning, replacement, deletion, and insertion. These diverse manipulations are all achieved through a shared set of varying inputs, processed by identical position control and appearance maintenance modules. The position control module projects the given 3D bounding box while preserving depth information and hierarchically injects it into the diffusion process, enabling precise control over object position and orientation. The appearance maintenance module preserves consistent attributes with a single reference image by employing a three-tiered approach: low-level detail preservation, high-level semantic maintenance, and the integration of 3D priors from a novel view synthesis model. Extensive qualitative and quantitative evaluations on the nuScenes dataset demonstrate DriveEditor's exceptional fidelity and controllability in generating diverse driving scene edits, as well as its remarkable ability to facilitate downstream tasks. Project page: https://yvanliang.github.io/DriveEditor.