COS3D: Collaborative Open-Vocabulary 3D Segmentation

TL;DR

COS3D introduces a collaborative framework for open-vocabulary 3D segmentation that effectively combines language and segmentation cues, achieving superior performance and broad applicability in 3D understanding tasks.

Contribution

The paper proposes a novel collaborative field concept and a two-stage training strategy to enhance open-vocabulary 3D segmentation performance.

Findings

Outperforms existing methods on benchmark datasets

Demonstrates versatility in applications like image-based 3D segmentation and robotics

Achieves high-quality prompt-segmentation inference through adaptive refinement

Abstract

Open-vocabulary 3D segmentation is a fundamental yet challenging task, requiring a mutual understanding of both segmentation and language. However, existing Gaussian-splatting-based methods rely either on a single 3D language field, leading to inferior segmentation, or on pre-computed class-agnostic segmentations, suffering from error accumulation. To address these limitations, we present COS3D, a new collaborative prompt-segmentation framework that contributes to effectively integrating complementary language and segmentation cues throughout its entire pipeline. We first introduce the new concept of collaborative field, comprising an instance field and a language field, as the cornerstone for collaboration. During training, to effectively construct the collaborative field, our key idea is to capture the intrinsic relationship between the instance field and language field, through a novel instance-to-language feature mapping and designing an efficient two-stage training strategy. During inference, to bridge distinct characteristics of the two fields, we further design an adaptive language-to-instance prompt refinement, promoting high-quality prompt-segmentation inference. Extensive experiments not only demonstrate COS3D's leading performance over existing methods on two widely-used benchmarks but also show its high potential to various applications,~\ie, novel image-based 3D segmentation, hierarchical segmentation, and robotics. The code is publicly available at \href{https://github.com/Runsong123/COS3D}{https://github.com/Runsong123/COS3D}.