DiSCO-3D : Discovering and segmenting Sub-Concepts from Open-vocabulary queries in NeRF

TL;DR

DiSCO-3D introduces a novel method for 3D semantic segmentation that combines unsupervised learning with open-vocabulary guidance, enabling discovery of sub-concepts in scenes based on user queries.

Contribution

It is the first approach to address 3D open-vocabulary sub-concept discovery by integrating neural fields with weak guidance and unsupervised segmentation.

Findings

Achieves effective open-vocabulary sub-concept discovery in 3D scenes.

Outperforms existing methods in open-vocabulary and unsupervised segmentation tasks.

Demonstrates state-of-the-art results in challenging edge cases.

Abstract

3D semantic segmentation provides high-level scene understanding for applications in robotics, autonomous systems, \textit{etc}. Traditional methods adapt exclusively to either task-specific goals (open-vocabulary segmentation) or scene content (unsupervised semantic segmentation). We propose DiSCO-3D, the first method addressing the broader problem of 3D Open-Vocabulary Sub-concepts Discovery, which aims to provide a 3D semantic segmentation that adapts to both the scene and user queries. We build DiSCO-3D on Neural Fields representations, combining unsupervised segmentation with weak open-vocabulary guidance. Our evaluations demonstrate that DiSCO-3D achieves effective performance in Open-Vocabulary Sub-concepts Discovery and exhibits state-of-the-art results in the edge cases of both open-vocabulary and unsupervised segmentation.