OpenGS-Fusion: Open-Vocabulary Dense Mapping with Hybrid 3D Gaussian Splatting for Refined Object-Level Understanding

TL;DR

OpenGS-Fusion introduces a novel open-vocabulary dense mapping framework that combines 3D Gaussian representations with SDFs and a multimodal language-guided approach to enhance 3D object understanding and scene reconstruction.

Contribution

It presents a new framework integrating 3D Gaussian and SDF for semantic fusion and a language-guided adaptive thresholding method for improved object segmentation.

Findings

Achieves 17% improvement in 3D mIoU over fixed threshold methods.

Outperforms existing methods in 3D object understanding.

Enhances scene reconstruction quality and language-guided interaction.

Abstract

Recent advancements in 3D scene understanding have made significant strides in enabling interaction with scenes using open-vocabulary queries, particularly for VR/AR and robotic applications. Nevertheless, existing methods are hindered by rigid offline pipelines and the inability to provide precise 3D object-level understanding given open-ended queries. In this paper, we present OpenGS-Fusion, an innovative open-vocabulary dense mapping framework that improves semantic modeling and refines object-level understanding. OpenGS-Fusion combines 3D Gaussian representation with a Truncated Signed Distance Field to facilitate lossless fusion of semantic features on-the-fly. Furthermore, we introduce a novel multimodal language-guided approach named MLLM-Assisted Adaptive Thresholding, which refines the segmentation of 3D objects by adaptively adjusting similarity thresholds, achieving an improvement 17\% in 3D mIoU compared to the fixed threshold strategy. Extensive experiments demonstrate that our method outperforms existing methods in 3D object understanding and scene reconstruction quality, as well as showcasing its effectiveness in language-guided scene interaction. The code is available at https://young-bit.github.io/opengs-fusion.github.io/ .