Segment Any 3D-Part in a Scene from a Sentence

TL;DR

This paper introduces a new large-scale 3D dataset with dense part annotations and a novel 3D-input framework for segmenting any 3D part in a scene based on natural language, advancing 3D scene understanding.

Contribution

It presents the first large-scale 3D part dataset and a 3D-input-only method for open-vocabulary part segmentation in scenes.

Findings

The 3D-PU dataset enables detailed part-level understanding.

OpenPart3D outperforms existing methods in open-vocabulary segmentation.

The approach generalizes well across different 3D scene datasets.

Abstract

This paper aims to achieve the segmentation of any 3D part in a scene based on natural language descriptions, extending beyond traditional object-level 3D scene understanding and addressing both data and methodological challenges. Due to the expensive acquisition and annotation burden, existing datasets and methods are predominantly limited to object-level comprehension. To overcome the limitations of data and annotation availability, we introduce the 3D-PU dataset, the first large-scale 3D dataset with dense part annotations, created through an innovative and cost-effective method for constructing synthetic 3D scenes with fine-grained part-level annotations, paving the way for advanced 3D-part scene understanding. On the methodological side, we propose OpenPart3D, a 3D-input-only framework to effectively tackle the challenges of part-level segmentation. Extensive experiments demonstrate the superiority of our approach in open-vocabulary 3D scene understanding tasks at the part level, with strong generalization capabilities across various 3D scene datasets.