# Integrating SAM Supervision for 3D Weakly Supervised Point Cloud Segmentation

## TL;DR

This paper introduces a novel method that leverages 2D foundation models to enhance 3D weakly supervised point cloud segmentation by propagating 2D masks into 3D space and using confidence-based regularization.

## Contribution

It proposes integrating 2D foundation model segmentation masks with 3D data, propagating labels via geometric correspondences, and applying confidence regularization to improve 3D segmentation with limited annotations.

## Key findings

- Significant improvement in 3D segmentation accuracy with sparse annotations.
- Effective propagation of 2D masks into 3D space enhances label coverage.
- Confidence-based filtering improves pseudo-label reliability.

## Abstract

Current methods for 3D semantic segmentation propose training models with limited annotations to address the difficulty of annotating large, irregular, and unordered 3D point cloud data. They usually focus on the 3D domain only, without leveraging the complementary nature of 2D and 3D data. Besides, some methods extend original labels or generate pseudo labels to guide the training, but they often fail to fully use these labels or address the noise within them. Meanwhile, the emergence of comprehensive and adaptable foundation models has offered effective solutions for segmenting 2D data. Leveraging this advancement, we present a novel approach that maximizes the utility of sparsely available 3D annotations by incorporating segmentation masks generated by 2D foundation models. We further propagate the 2D segmentation masks into the 3D space by establishing geometric correspondences between 3D scenes and 2D views. We extend the highly sparse annotations to encompass the areas delineated by 3D masks, thereby substantially augmenting the pool of available labels. Furthermore, we apply confidence- and uncertainty-based consistency regularization on augmentations of the 3D point cloud and select the reliable pseudo labels, which are further spread on the 3D masks to generate more labels. This innovative strategy bridges the gap between limited 3D annotations and the powerful capabilities of 2D foundation models, ultimately improving the performance of 3D weakly supervised segmentation.