Exploring the Untouched Sweeps for Conflict-Aware 3D Segmentation Pretraining

TL;DR

This paper introduces a novel pretraining method for LiDAR-camera 3D perception that leverages unexplored frame pairs and conflict-aware contrastive loss, significantly improving segmentation performance across multiple datasets.

Contribution

It proposes a VFM-driven sample exploring module and a conflict-aware contrastive loss to utilize more data and maintain semantic consistency during pretraining.

Findings

Achieved over 3% improvement in mIoU on three datasets.

Enhanced generalization to different 3D backbones.

Effectively utilizes unpaired frames and semantic priors.

Abstract

LiDAR-camera 3D representation pretraining has shown significant promise for 3D perception tasks and related applications. However, two issues widely exist in this framework: 1) Solely keyframes are used for training. For example, in nuScenes, a substantial quantity of unpaired LiDAR and camera frames remain unutilized, limiting the representation capabilities of the pretrained network. 2) The contrastive loss erroneously distances points and image regions with identical semantics but from different frames, disturbing the semantic consistency of the learned presentations. In this paper, we propose a novel Vision-Foundation-Model-driven sample exploring module to meticulously select LiDAR-Image pairs from unexplored frames, enriching the original training set. We utilized timestamps and the semantic priors from VFMs to identify well-synchronized training pairs and to discover samples with diverse content. Moreover, we design a cross- and intra-modal conflict-aware contrastive loss using the semantic mask labels of VFMs to avoid contrasting semantically similar points and image regions. Our method consistently outperforms existing state-of-the-art pretraining frameworks across three major public autonomous driving datasets: nuScenes, SemanticKITTI, and Waymo on 3D semantic segmentation by +3.0\%, +3.0\%, and +3.3\% in mIoU, respectively. Furthermore, our approach exhibits adaptable generalization to different 3D backbones and typical semantic masks generated by non-VFM models.