SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Instance Segmentation

TL;DR

SAMPro3D introduces a zero-shot 3D instance segmentation method that applies the pretrained Segment Anything Model to 2D views, aligning prompts across views for consistent 3D segmentation without additional training.

Contribution

It proposes a novel approach to locate SAM prompts in 3D space and align them across multiple views, enhancing 3D segmentation performance without extra training.

Findings

Achieves comparable or better performance than previous methods.

Surpasses human annotations in many cases.

Provides a new fine-grained 3D dataset, ScanNet200-Fine50.

Abstract

We introduce SAMPro3D for zero-shot instance segmentation of 3D scenes. Given the 3D point cloud and multiple posed RGB-D frames of 3D scenes, our approach segments 3D instances by applying the pretrained Segment Anything Model (SAM) to 2D frames. Our key idea involves locating SAM prompts in 3D to align their projected pixel prompts across frames, ensuring the view consistency of SAM-predicted masks. Moreover, we suggest selecting prompts from the initial set guided by the information of SAM-predicted masks across all views, which enhances the overall performance. We further propose to consolidate different prompts if they are segmenting different surface parts of the same 3D instance, bringing a more comprehensive segmentation. Notably, our method does not require any additional training. Extensive experiments on diverse benchmarks show that our method achieves comparable or better performance compared to previous zero-shot or fully supervised approaches, and in many cases surpasses human annotations. Furthermore, since our fine-grained predictions often lack annotations in available datasets, we present ScanNet200-Fine50 test data which provides fine-grained annotations on 50 scenes from ScanNet200 dataset. The project page can be accessed at https://mutianxu.github.io/sampro3d/.