GLS: Geometry-aware 3D Language Gaussian Splatting

TL;DR

GLS is a unified framework that enhances 3D surface reconstruction and open-vocabulary segmentation by integrating geometric cues and CLIP features, achieving state-of-the-art results on multiple datasets.

Contribution

It introduces a novel unified approach combining 3D Gaussian Splatting with geometric and semantic cues for improved 3D reconstruction and segmentation.

Findings

Outperforms existing methods on MuSHRoom, ScanNet++, and LERF-OVS datasets.

Effectively combines geometric cues with CLIP features for better surface and semantic accuracy.

Demonstrates joint optimization benefits for 3D reconstruction and segmentation tasks.

Abstract

Recently, 3D Gaussian Splatting (3DGS) has achieved impressive performance on indoor surface reconstruction and 3D open-vocabulary segmentation. This paper presents GLS, a unified framework of 3D surface reconstruction and open-vocabulary segmentation based on 3DGS. GLS extends two fields by improving their sharpness and smoothness. For indoor surface reconstruction, we introduce surface normal prior as a geometric cue to guide the rendered normal, and use the normal error to optimize the rendered depth. For 3D open-vocabulary segmentation, we employ 2D CLIP features to guide instance features and enhance the surface smoothness, then utilize DEVA masks to maintain their view consistency. Extensive experiments demonstrate the effectiveness of jointly optimizing surface reconstruction and 3D open-vocabulary segmentation, where GLS surpasses state-of-the-art approaches of each task on MuSHRoom, ScanNet++ and LERF-OVS datasets. Project webpage: https://jiaxiongq.github.io/GLS_ProjectPage.