Hi-LSplat: Hierarchical 3D Language Gaussian Splatting

TL;DR

Hi-LSplat introduces a view-consistent hierarchical approach for 3D language Gaussian splatting, effectively capturing complex semantic hierarchies and improving open-vocabulary 3D scene understanding.

Contribution

The paper presents a novel hierarchical semantic framework that lifts 2D features to 3D, ensuring view consistency and hierarchical semantic understanding in 3D language modeling.

Findings

Outperforms existing methods in 3D open-vocabulary segmentation.

Effectively captures hierarchical semantic structures in 3D scenes.

Demonstrates robustness across hierarchical semantic datasets.

Abstract

Modeling 3D language fields with Gaussian Splatting for open-ended language queries has recently garnered increasing attention. However, recent 3DGS-based models leverage view-dependent 2D foundation models to refine 3D semantics but lack a unified 3D representation, leading to view inconsistencies. Additionally, inherent open-vocabulary challenges cause inconsistencies in object and relational descriptions, impeding hierarchical semantic understanding. In this paper, we propose Hi-LSplat, a view-consistent Hierarchical Language Gaussian Splatting work for 3D open-vocabulary querying. To achieve view-consistent 3D hierarchical semantics, we first lift 2D features to 3D features by constructing a 3D hierarchical semantic tree with layered instance clustering, which addresses the view inconsistency issue caused by 2D semantic features. Besides, we introduce instance-wise and part-wise contrastive losses to capture all-sided hierarchical semantic representations. Notably, we construct two hierarchical semantic datasets to better assess the model's ability to distinguish different semantic levels. Extensive experiments highlight our method's superiority in 3D open-vocabulary segmentation and localization. Its strong performance on hierarchical semantic datasets underscores its ability to capture complex hierarchical semantics within 3D scenes.