ReferSplat: Referring Segmentation in 3D Gaussian Splatting

TL;DR

ReferringSplat introduces a novel 3D segmentation task based on natural language, supported by a new dataset, and proposes a framework that significantly advances 3D multi-modal understanding and spatial reasoning.

Contribution

The paper presents the first R3DGS dataset and a new framework, ReferSplat, for natural language-based 3D object segmentation, achieving state-of-the-art results.

Findings

RefSplat outperforms previous methods on R3DGS and open-vocabulary benchmarks.

The new dataset enables research on language-guided 3D segmentation.

Explicit modeling of spatial relationships improves 3D multi-modal understanding.

Abstract

We introduce Referring 3D Gaussian Splatting Segmentation (R3DGS), a new task that aims to segment target objects in a 3D Gaussian scene based on natural language descriptions, which often contain spatial relationships or object attributes. This task requires the model to identify newly described objects that may be occluded or not directly visible in a novel view, posing a significant challenge for 3D multi-modal understanding. Developing this capability is crucial for advancing embodied AI. To support research in this area, we construct the first R3DGS dataset, Ref-LERF. Our analysis reveals that 3D multi-modal understanding and spatial relationship modeling are key challenges for R3DGS. To address these challenges, we propose ReferSplat, a framework that explicitly models 3D Gaussian points with natural language expressions in a spatially aware paradigm. ReferSplat achieves state-of-the-art performance on both the newly proposed R3DGS task and 3D open-vocabulary segmentation benchmarks. Dataset and code are available at https://github.com/heshuting555/ReferSplat.