Leveraging Large-Scale Pretrained Vision Foundation Models for Label-Efficient 3D Point Cloud Segmentation

TL;DR

This paper introduces a framework that leverages large-scale pre-trained vision models to improve 3D point cloud segmentation by projecting 2D masks into 3D space and fusing labels, enabling label-efficient 3D scene understanding.

Contribution

It adapts large 2D foundation models for 3D segmentation by projecting 2D masks into 3D and fusing labels, a novel approach for label-efficient 3D scene analysis.

Findings

Effective 3D segmentation using 2D foundation models.

Robust pseudo labels through semantic label fusion.

Successful application on ScanNet dataset.

Abstract

Recently, large-scale pre-trained models such as Segment-Anything Model (SAM) and Contrastive Language-Image Pre-training (CLIP) have demonstrated remarkable success and revolutionized the field of computer vision. These foundation vision models effectively capture knowledge from a large-scale broad data with their vast model parameters, enabling them to perform zero-shot segmentation on previously unseen data without additional training. While they showcase competence in 2D tasks, their potential for enhancing 3D scene understanding remains relatively unexplored. To this end, we present a novel framework that adapts various foundational models for the 3D point cloud segmentation task. Our approach involves making initial predictions of 2D semantic masks using different large vision models. We then project these mask predictions from various frames of RGB-D video sequences into 3D space. To generate robust 3D semantic pseudo labels, we introduce a semantic label fusion strategy that effectively combines all the results via voting. We examine diverse scenarios, like zero-shot learning and limited guidance from sparse 2D point labels, to assess the pros and cons of different vision foundation models. Our approach is experimented on ScanNet dataset for 3D indoor scenes, and the results demonstrate the effectiveness of adopting general 2D foundation models on solving 3D point cloud segmentation tasks.