CLAP: Unsupervised 3D Representation Learning for Fusion 3D Perception via Curvature Sampling and Prototype Learning

TL;DR

CLAP introduces a joint unsupervised pre-training approach for images and point clouds that leverages curvature sampling and prototype learning to enhance 3D perception, significantly outperforming previous methods.

Contribution

The paper proposes a novel differentiable-rendering-based pre-training method that jointly learns from images and point clouds using curvature sampling and learnable prototypes.

Findings

Achieves up to 100% performance improvement over previous SOTA methods.

Effectively exploits the complementarity of image semantics and 3D structure.

Demonstrates strong results on NuScenes and Waymo datasets.

Abstract

Unsupervised 3D representation learning reduces the burden of labeling multimodal 3D data for fusion perception tasks. Among different pre-training paradigms, differentiable-rendering-based methods have shown most promise. However, existing works separately conduct pre-training for each modalities due to computational costs of processing large point clouds with images. As such, mutual benefit of high-level semantics (from image) and 3D structure (from point cloud) has not been exploited. To address this gap, we propose a joint unsupervised differentiable-rendering-based pre-training method for images and point clouds, termed CLAP, short for Curvature sampLing and leArnable Prototype. Specifically, our method overcomes the computational hurdle by Curvature Sampling to select the more informative points/pixels for pre-training. To uncover the performance benefits brought by their complementarity, we propose to use learnable prototypes to represent parts of the 3D scenes in a common feature space and an Expectation-Maximization training scheme to associate embeddings of each modality to prototypes. We further propose a swapping prediction loss that explores their interplay through prototypes along with a Gram Matrix Regularization term to maintain training stability. Experiments on NuScenes and Waymo datasets show that CLAP achieves up to 100% more performance gain as compared to previous SOTA pre-training methods.