OpenSplat3D: Open-Vocabulary 3D Instance Segmentation using Gaussian Splatting

TL;DR

OpenSplat3D introduces a novel open-vocabulary 3D instance segmentation method that combines Gaussian Splatting with vision-language models, enabling flexible, text-driven scene understanding without manual labels.

Contribution

We propose a new approach that extends 3D Gaussian Splatting for open-vocabulary instance segmentation using language embeddings and contrastive learning, without manual annotations.

Findings

Effective segmentation on ScanNet++ validation set

Achieves open-vocabulary object identification

Outperforms baseline methods in scene understanding

Abstract

3D Gaussian Splatting (3DGS) has emerged as a powerful representation for neural scene reconstruction, offering high-quality novel view synthesis while maintaining computational efficiency. In this paper, we extend the capabilities of 3DGS beyond pure scene representation by introducing an approach for open-vocabulary 3D instance segmentation without requiring manual labeling, termed OpenSplat3D. Our method leverages feature-splatting techniques to associate semantic information with individual Gaussians, enabling fine-grained scene understanding. We incorporate Segment Anything Model instance masks with a contrastive loss formulation as guidance for the instance features to achieve accurate instance-level segmentation. Furthermore, we utilize language embeddings of a vision-language model, allowing for flexible, text-driven instance identification. This combination enables our system to identify and segment arbitrary objects in 3D scenes based on natural language descriptions. We show results on LERF-mask and LERF-OVS as well as the full ScanNet++ validation set, demonstrating the effectiveness of our approach.