OE3DIS: Open-Ended 3D Point Cloud Instance Segmentation

TL;DR

This paper introduces OE-3DIS, a new approach for open-ended 3D point cloud instance segmentation that does not require predefined class names, improving flexibility and performance over existing methods.

Contribution

The paper proposes OE-3DIS, a novel problem setting eliminating class name dependence, along with strong baselines and a new open-ended score for evaluating 3D segmentation quality.

Findings

OE-3DIS outperforms baselines on ScanNet datasets.

The method surpasses state-of-the-art OV-3DIS in open-ended settings.

Significant improvements in semantic and geometric mask quality.

Abstract

Open-Vocab 3D Instance Segmentation methods (OV-3DIS) have recently demonstrated their ability to generalize to unseen objects. However, these methods still depend on predefined class names during testing, restricting the autonomy of agents. To mitigate this constraint, we propose a novel problem termed Open-Ended 3D Instance Segmentation (OE-3DIS), which eliminates the necessity for predefined class names during testing. Moreover, we contribute a comprehensive set of strong baselines, derived from OV-3DIS approaches and leveraging 2D Multimodal Large Language Models. To assess the performance of our OE-3DIS system, we introduce a novel Open-Ended score, evaluating both the semantic and geometric quality of predicted masks and their associated class names, alongside the standard AP score. Our approach demonstrates significant performance improvements over the baselines on the ScanNet200 and ScanNet++ datasets. Remarkably, our method surpasses the performance of Open3DIS, the current state-of-the-art method in OV-3DIS, even in the absence of ground-truth object class names.