Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models

TL;DR

This paper introduces Diff2Scene, a novel approach using pre-trained diffusion models for open-vocabulary 3D semantic segmentation that eliminates the need for labeled 3D data and outperforms existing methods.

Contribution

Diff2Scene leverages frozen text-image generative model representations with masks for 3D segmentation, enabling open-vocabulary understanding without labeled 3D data.

Findings

Outperforms baseline methods in 3D semantic segmentation

Achieves 12% improvement on ScanNet200

Effectively identifies objects and materials in 3D scenes

Abstract

In this paper, we investigate the use of diffusion models which are pre-trained on large-scale image-caption pairs for open-vocabulary 3D semantic understanding. We propose a novel method, namely Diff2Scene, which leverages frozen representations from text-image generative models, along with salient-aware and geometric-aware masks, for open-vocabulary 3D semantic segmentation and visual grounding tasks. Diff2Scene gets rid of any labeled 3D data and effectively identifies objects, appearances, materials, locations and their compositions in 3D scenes. We show that it outperforms competitive baselines and achieves significant improvements over state-of-the-art methods. In particular, Diff2Scene improves the state-of-the-art method on ScanNet200 by 12%.