LTDA-Drive: LLMs-guided Generative Models based Long-tail Data Augmentation for Autonomous Driving

TL;DR

LTDA-Drive is a novel LLM-guided data augmentation framework that synthesizes diverse, high-quality long-tail samples to improve 3D perception in autonomous driving, especially for rare and safety-critical classes.

Contribution

It introduces a three-stage process using diffusion models and LLMs to generate and filter synthetic tail-class data, addressing scarcity and diversity issues in long-tail data augmentation.

Findings

Significantly improves tail-class detection performance.

Achieves 34.75% improvement for rare classes on KITTI dataset.

Demonstrates effectiveness of LLM-guided augmentation in autonomous driving.

Abstract

3D perception plays an essential role for improving the safety and performance of autonomous driving. Yet, existing models trained on real-world datasets, which naturally exhibit long-tail distributions, tend to underperform on rare and safety-critical, vulnerable classes, such as pedestrians and cyclists. Existing studies on reweighting and resampling techniques struggle with the scarcity and limited diversity within tail classes. To address these limitations, we introduce LTDA-Drive, a novel LLM-guided data augmentation framework designed to synthesize diverse, high-quality long-tail samples. LTDA-Drive replaces head-class objects in driving scenes with tail-class objects through a three-stage process: (1) text-guided diffusion models remove head-class objects, (2) generative models insert instances of the tail classes, and (3) an LLM agent filters out low-quality synthesized images. Experiments conducted on the KITTI dataset show that LTDA-Drive significantly improves tail-class detection, achieving 34.75\% improvement for rare classes over counterpart methods. These results further highlight the effectiveness of LTDA-Drive in tackling long-tail challenges by generating high-quality and diverse data.