Cross-Modal Self-Supervised Learning with Effective Contrastive Units for LiDAR Point Clouds

TL;DR

This paper introduces a cross-modal contrastive learning approach for LiDAR point clouds, leveraging camera data to improve 3D perception tasks in autonomous driving, with novel contrastive units tailored for LiDAR data.

Contribution

It systematically studies multi-modality contrastive learning and proposes instance-aware, similarity-balanced units specifically designed for LiDAR point cloud pre-training.

Findings

Cross-modality contrastive learning outperforms single and multi-modality methods.

Proposed units significantly improve 3D detection and segmentation accuracy.

Achieves state-of-the-art results on multiple autonomous driving benchmarks.

Abstract

3D perception in LiDAR point clouds is crucial for a self-driving vehicle to properly act in 3D environment. However, manually labeling point clouds is hard and costly. There has been a growing interest in self-supervised pre-training of 3D perception models. Following the success of contrastive learning in images, current methods mostly conduct contrastive pre-training on point clouds only. Yet an autonomous driving vehicle is typically supplied with multiple sensors including cameras and LiDAR. In this context, we systematically study single modality, cross-modality, and multi-modality for contrastive learning of point clouds, and show that cross-modality wins over other alternatives. In addition, considering the huge difference between the training sources in 2D images and 3D point clouds, it remains unclear how to design more effective contrastive units for LiDAR. We therefore propose the instance-aware and similarity-balanced contrastive units that are tailored for self-driving point clouds. Extensive experiments reveal that our approach achieves remarkable performance gains over various point cloud models across the downstream perception tasks of LiDAR based 3D object detection and 3D semantic segmentation on the four popular benchmarks including Waymo Open Dataset, nuScenes, SemanticKITTI and ONCE.