DriveCtrl: Conditioned Sim-to-Real Driving Video Generation

TL;DR

DriveCtrl is a depth-conditioned framework that generates realistic, temporally coherent driving videos from simulation, matching real-world style and preserving annotations for autonomous driving applications.

Contribution

It introduces a structure-aware adapter for depth-guided, controllable video synthesis and a scalable pipeline for transforming simulator videos into realistic driving footage.

Findings

DriveCtrl outperforms baseline models in realism and temporal consistency.

The generated videos improve perception task performance.

The proposed DVRS metric effectively assesses driving video realism.

Abstract

Large-scale labelled driving video data is essential for training autonomous driving systems. Although simulation offers scalable and fully annotated data, the domain gap between synthetic and real-world driving videos significantly limits its utility for downstream deployment. Existing video generation methods are not well-suited for this task, as they fail to simultaneously preserve scene structure, object dynamics, temporal consistency, and visual realism, all of which are critical for maintaining annotation validity in generated data. In this paper, we present DriveCtrl, a depth-conditioned controllable sim-to-real video generation framework for realistic driving video synthesis. Built upon a pretrained video foundation model, DriveCtrl introduces a structure-aware adapter that enables depth-guided generation while preserving the scene layout and motion patterns of the source simulation, producing temporally coherent driving videos that remain aligned with the original simulated sequences. We further introduce a scalable data generation pipeline that transforms simulator videos into realistic driving footage matching the visual style of a target real-world dataset. The pipeline supports three conditioning signals: structural depth, reference-dataset style, and text prompts, while preserving frame-level annotations for downstream perception tasks. To better assess this task, we propose a driving-domain-specific knowledge-informed evaluation metric called Driving Video Realism Score (DVRS) that assesses the realism of generated videos. Experiments demonstrate that DriveCtrl consistently outperforms the base model and competing alternatives in realism, temporal quality, and perception task performance, substantially narrowing the sim-to-real gap for driving video generation.