CLLAP: Contrastive Learning-based LiDAR-Augmented Pretraining for Enhanced Radar-Camera Fusion

TL;DR

CLLAP introduces a contrastive learning framework that uses LiDAR data to generate pseudo-radar data, improving 3D object detection in autonomous driving by enhancing radar-camera fusion models.

Contribution

It proposes a novel LiDAR-to-Radar sampling method and a dual-stage contrastive learning strategy for effective self-supervised pretraining of fusion models.

Findings

Significant performance improvements on NuScenes and Lyft datasets.

Enhanced feature extraction capabilities of baseline models.

Improved detection accuracy and robustness in adverse conditions.

Abstract

Accurate 3D object detection is critical for autonomous driving, necessitating reliable, cost-effective sensors capable of operating in adverse weather conditions. Camera and millimeter-wave radar fusion has emerged as a promising solution; however, these methods often rely on finely annotated radar data, which is scarce and labor-intensive to produce. To address this challenge, we present CLLAP, a Contrastive Learning-based LiDAR-Augmented Pretraining framework that enhances the performance of existing radar-camera fusion methods for 3D object detection. CLLAP leverages abundant LiDAR data to generate pseudo-radar data using the proposed L2R (LiDAR-to-Radar) Sampling method. Then, it incorporates this data into a novel dual-stage, dual-modality contrastive learning strategy, enabling effective self-supervised learning from paired pseudo-radar and image data. This approach facilitates effective pretraining of existing radar-camera fusion models in a plug-and-play manner, enhancing their feature extraction capabilities and improving detection accuracy and robustness. Experimental results using NuScenes and Lyft Level 5 datasets demonstrate significant performance improvements across three baseline models, highlighting CLLAP's effectiveness in advancing radar-camera fusion for autonomous driving applications.