Details Matter for Indoor Open-vocabulary 3D Instance Segmentation

TL;DR

This paper introduces a novel approach for open-vocabulary 3D instance segmentation that combines multiple concepts, refines them, and employs a new filtering method, achieving state-of-the-art results on benchmark datasets.

Contribution

The paper presents a comprehensive recipe that integrates existing concepts with new refinements, including Alpha-CLIP and SMS score, to significantly improve OV-3DIS performance.

Findings

Achieves state-of-the-art results on ScanNet200 and S3DIS datasets.

Outperforms end-to-end closed-vocabulary methods.

Effective proposal filtering and classification enhancements.

Abstract

Unlike closed-vocabulary 3D instance segmentation that is often trained end-to-end, open-vocabulary 3D instance segmentation (OV-3DIS) often leverages vision-language models (VLMs) to generate 3D instance proposals and classify them. While various concepts have been proposed from existing research, we observe that these individual concepts are not mutually exclusive but complementary. In this paper, we propose a new state-of-the-art solution for OV-3DIS by carefully designing a recipe to combine the concepts together and refining them to address key challenges. Our solution follows the two-stage scheme: 3D proposal generation and instance classification. We employ robust 3D tracking-based proposal aggregation to generate 3D proposals and remove overlapped or partial proposals by iterative merging/removal. For the classification stage, we replace the standard CLIP model with Alpha-CLIP, which incorporates object masks as an alpha channel to reduce background noise and obtain object-centric representation. Additionally, we introduce the standardized maximum similarity (SMS) score to normalize text-to-proposal similarity, effectively filtering out false positives and boosting precision. Our framework achieves state-of-the-art performance on ScanNet200 and S3DIS across all AP and AR metrics, even surpassing an end-to-end closed-vocabulary method.