Learning to Adapt SAM for Segmenting Cross-domain Point Clouds

TL;DR

This paper introduces a novel approach that leverages the SAM foundation model and image correspondence to improve unsupervised domain adaptation in 3D point cloud segmentation, achieving state-of-the-art results.

Contribution

It proposes a hybrid feature augmentation method that aligns 3D point cloud features with SAM's features using associated images, addressing domain discrepancies.

Findings

Achieves state-of-the-art performance on multiple datasets.

Effectively aligns 3D features with SAM's features across domains.

Enhances 3D segmentation robustness in cross-domain scenarios.

Abstract

Unsupervised domain adaptation (UDA) in 3D segmentation tasks presents a formidable challenge, primarily stemming from the sparse and unordered nature of point cloud data. Especially for LiDAR point clouds, the domain discrepancy becomes obvious across varying capture scenes, fluctuating weather conditions, and the diverse array of LiDAR devices in use. While previous UDA methodologies have often sought to mitigate this gap by aligning features between source and target domains, this approach falls short when applied to 3D segmentation due to the substantial domain variations. Inspired by the remarkable generalization capabilities exhibited by the vision foundation model, SAM, in the realm of image segmentation, our approach leverages the wealth of general knowledge embedded within SAM to unify feature representations across diverse 3D domains and further solves the 3D domain adaptation problem. Specifically, we harness the corresponding images associated with point clouds to facilitate knowledge transfer and propose an innovative hybrid feature augmentation methodology, which significantly enhances the alignment between the 3D feature space and SAM's feature space, operating at both the scene and instance levels. Our method is evaluated on many widely-recognized datasets and achieves state-of-the-art performance.