DCSEG: Decoupled 3D Open-Set Segmentation using Gaussian Splatting

TL;DR

DCSEG introduces a modular 3D segmentation framework that decouples scene reconstruction from semantic labeling, leveraging Gaussian splatting and 2D models for flexible, open-vocabulary scene understanding in robotics and AR/VR.

Contribution

It proposes a decoupled 3D segmentation pipeline that integrates Gaussian splatting with 2D open-vocabulary models, enabling flexible and modular scene segmentation without retraining 3D representations.

Findings

Outperforms NeRF-based methods on synthetic and real datasets in mIoU and mAcc.

Demonstrates robustness in challenging and long-tail classes.

Shows modularity by varying 2D backbones affects segmentation results.

Abstract

Open-set 3D segmentation represents a major point of interest for multiple downstream robotics and augmented/virtual reality applications. We present a decoupled 3D segmentation pipeline to ensure modularity and adaptability to novel 3D representations as well as semantic segmentation foundation models. We first reconstruct a scene with 3D Gaussians and learn class-agnostic features through contrastive supervision from a 2D instance proposal network. These 3D features are then clustered to form coarse object- or part-level masks. Finally, we match each 3D cluster to class-aware masks predicted by a 2D open-vocabulary segmentation model, assigning semantic labels without retraining the 3D representation. Our decoupled design (1) provides a plug-and-play interface for swapping different 2D or 3D modules, (2) ensures multi-object instance segmentation at no extra cost, and (3) leverages rich 3D geometry for robust scene understanding. We evaluate on synthetic and real-world indoor datasets, demonstrating improved performance over comparable NeRF-based pipelines on mIoU and mAcc, particularly for challenging or long-tail classes. We also show how varying the 2D backbone affects the final segmentation, highlighting the modularity of our framework. These results confirm that decoupling 3D mask proposal and semantic classification can deliver flexible, efficient, and open-vocabulary 3D segmentation.