Taking Language Embedded 3D Gaussian Splatting into the Wild

TL;DR

This paper introduces a novel framework extending language embedded 3D Gaussian splatting for open-vocabulary scene understanding from unconstrained photo collections, enabling immersive 3D architectural analysis and editing.

Contribution

It proposes a new method combining multi-appearance features, uncertainty maps, and a transient uncertainty-aware autoencoder for improved 3D scene understanding and segmentation.

Findings

Outperforms existing methods in open-vocabulary segmentation

Enables interactive 3D scene exploration and editing

Introduces PT-OVS benchmark dataset for evaluation

Abstract

Recent advances in leveraging large-scale Internet photo collections for 3D reconstruction have enabled immersive virtual exploration of landmarks and historic sites worldwide. However, little attention has been given to the immersive understanding of architectural styles and structural knowledge, which remains largely confined to browsing static text-image pairs. Therefore, can we draw inspiration from 3D in-the-wild reconstruction techniques and use unconstrained photo collections to create an immersive approach for understanding the 3D structure of architectural components? To this end, we extend language embedded 3D Gaussian splatting (3DGS) and propose a novel framework for open-vocabulary scene understanding from unconstrained photo collections. Specifically, we first render multiple appearance images from the same viewpoint as the unconstrained image with the reconstructed radiance field, then extract multi-appearance CLIP features and two types of language feature uncertainty maps-transient and appearance uncertainty-derived from the multi-appearance features to guide the subsequent optimization process. Next, we propose a transient uncertainty-aware autoencoder, a multi-appearance language field 3DGS representation, and a post-ensemble strategy to effectively compress, learn, and fuse language features from multiple appearances. Finally, to quantitatively evaluate our method, we introduce PT-OVS, a new benchmark dataset for assessing open-vocabulary segmentation performance on unconstrained photo collections. Experimental results show that our method outperforms existing methods, delivering accurate open-vocabulary segmentation and enabling applications such as interactive roaming with open-vocabulary queries, architectural style pattern recognition, and 3D scene editing.