FastLGS: Speeding up Language Embedded Gaussians with Feature Grid Mapping

TL;DR

FastLGS enables real-time, high-resolution, open-vocabulary 3D scene understanding by integrating semantic feature grids with 3D Gaussian Splatting, achieving significant speed and accuracy improvements over existing methods.

Contribution

The paper introduces semantic feature grids for efficient multi-view CLIP feature storage and mapping in 3D Gaussian Splatting, enabling real-time open-vocabulary queries in 3D scenes.

Findings

FastLGS is 98x faster than LERF.

FastLGS outperforms state-of-the-art in speed and accuracy.

FastLGS is compatible with downstream 3D tasks like segmentation and inpainting.

Abstract

The semantically interactive radiance field has always been an appealing task for its potential to facilitate user-friendly and automated real-world 3D scene understanding applications. However, it is a challenging task to achieve high quality, efficiency and zero-shot ability at the same time with semantics in radiance fields. In this work, we present FastLGS, an approach that supports real-time open-vocabulary query within 3D Gaussian Splatting (3DGS) under high resolution. We propose the semantic feature grid to save multi-view CLIP features which are extracted based on Segment Anything Model (SAM) masks, and map the grids to low dimensional features for semantic field training through 3DGS. Once trained, we can restore pixel-aligned CLIP embeddings through feature grids from rendered features for open-vocabulary queries. Comparisons with other state-of-the-art methods prove that FastLGS can achieve the first place performance concerning both speed and accuracy, where FastLGS is 98x faster than LERF and 4x faster than LangSplat. Meanwhile, experiments show that FastLGS is adaptive and compatible with many downstream tasks, such as 3D segmentation and 3D object inpainting, which can be easily applied to other 3D manipulation systems.