DriveDiTFit: Fine-tuning Diffusion Transformers for Autonomous Driving

TL;DR

DriveDiTFit is a novel method that fine-tunes diffusion transformers to generate diverse, high-quality autonomous driving data efficiently, reducing the need for costly manual data collection across various weather and lighting conditions.

Contribution

It introduces a gap-driven modulation technique and an embedding module for weather and lighting diversity, enabling efficient fine-tuning of diffusion transformers for autonomous driving data generation.

Findings

Efficient generation of diverse real driving data.

High-quality small object and scenario synthesis.

Effective handling of weather and lighting variations.

Abstract

In autonomous driving, deep models have shown remarkable performance across various visual perception tasks with the demand of high-quality and huge-diversity training datasets. Such datasets are expected to cover various driving scenarios with adverse weather, lighting conditions and diverse moving objects. However, manually collecting these data presents huge challenges and expensive cost. With the rapid development of large generative models, we propose DriveDiTFit, a novel method for efficiently generating autonomous Driving data by Fine-tuning pre-trained Diffusion Transformers (DiTs). Specifically, DriveDiTFit utilizes a gap-driven modulation technique to carefully select and efficiently fine-tune a few parameters in DiTs according to the discrepancy between the pre-trained source data and the target driving data. Additionally, DriveDiTFit develops an effective weather and lighting condition embedding module to ensure diversity in the generated data, which is initialized by a nearest-semantic-similarity initialization approach. Through progressive tuning scheme to refined the process of detail generation in early diffusion process and enlarging the weights corresponding to small objects in training loss, DriveDiTFit ensures high-quality generation of small moving objects in the generated data. Extensive experiments conducted on driving datasets confirm that our method could efficiently produce diverse real driving data. The source codes will be available at https://github.com/TtuHamg/DriveDiTFit.