DrivePTS: A Progressive Learning Framework with Textual and Structural Enhancement for Driving Scene Generation

TL;DR

DrivePTS is a novel framework that enhances driving scene generation by using progressive learning, detailed multi-view textual descriptions, and frequency-guided structural loss to improve diversity, fidelity, and controllability.

Contribution

It introduces a progressive learning strategy with mutual information constraints, multi-view hierarchical textual guidance, and a frequency-guided loss for better scene synthesis.

Findings

Achieves state-of-the-art fidelity and controllability.

Successfully generates rare and complex driving scenes.

Improves structural and semantic details in generated scenes.

Abstract

Synthesis of diverse driving scenes serves as a crucial data augmentation technique for validating the robustness and generalizability of autonomous driving systems. Current methods aggregate high-definition (HD) maps and 3D bounding boxes as geometric conditions in diffusion models for conditional scene generation. However, implicit inter-condition dependency causes generation failures when control conditions change independently. Additionally, these methods suffer from insufficient details in both semantic and structural aspects. Specifically, brief and view-invariant captions restrict semantic contexts, resulting in weak background modeling. Meanwhile, the standard denoising loss with uniform spatial weighting neglects foreground structural details, causing visual distortions and blurriness. To address these challenges, we propose DrivePTS, which incorporates three key innovations. Firstly, our framework adopts a progressive learning strategy to mitigate inter-dependency between geometric conditions, reinforced by an explicit mutual information constraint. Secondly, a Vision-Language Model is utilized to generate multi-view hierarchical descriptions across six semantic aspects, providing fine-grained textual guidance. Thirdly, a frequency-guided structure loss is introduced to strengthen the model's sensitivity to high-frequency elements, improving foreground structural fidelity. Extensive experiments demonstrate that our DrivePTS achieves state-of-the-art fidelity and controllability in generating diverse driving scenes. Notably, DrivePTS successfully generates rare scenes where prior methods fail, highlighting its strong generalization ability.