GALA: Guided Attention with Language Alignment for Open Vocabulary Gaussian Splatting

TL;DR

GALA introduces a novel framework that combines guided attention with language alignment to enable open-vocabulary 3D scene understanding using Gaussian Splatting, effectively capturing fine-grained, language-aware 3D representations from 2D images.

Contribution

The paper proposes a cross-attention module with learnable codebooks for encoding view-independent semantic embeddings, enhancing open-vocabulary 3D scene understanding with reduced memory usage.

Findings

GALA achieves state-of-the-art open-vocabulary performance on real-world datasets.

It effectively supports 2D and 3D queries with improved semantic consistency.

The framework reduces memory consumption compared to previous methods.

Abstract

3D scene reconstruction and understanding have gained increasing popularity, yet existing methods still struggle to capture fine-grained, language-aware 3D representations from 2D images. In this paper, we present GALA, a novel framework for open-vocabulary 3D scene understanding with 3D Gaussian Splatting (3DGS). GALA distills a scene-specific 3D instance feature field via self-supervised contrastive learning. To extend to generalized language feature fields, we introduce the core contribution of GALA, a cross-attention module with two learnable codebooks that encode view-independent semantic embeddings. This design not only ensures intra-instance feature similarity but also supports seamless 2D and 3D open-vocabulary queries. It reduces memory consumption by avoiding per-Gaussian high-dimensional feature learning. Extensive experiments on real-world datasets demonstrate GALA's remarkable open-vocabulary performance on both 2D and 3D.