Gen-LangSplat: Generalized Language Gaussian Splatting with Pre-Trained Feature Compression

TL;DR

Gen-LangSplat introduces a scalable, scene-agnostic approach for 3D language field modeling by replacing scene-specific autoencoders with a pre-trained generalized autoencoder, enabling efficient open-vocabulary querying across diverse environments.

Contribution

It proposes a generalized autoencoder trained on large-scale data, removing the need for scene-specific training in 3D language field construction.

Findings

Achieves comparable or better querying performance than scene-specific methods.

Demonstrates efficient, scalable 3D language modeling without scene-specific training.

Validates the effectiveness of fixed latent space for open-vocabulary querying.

Abstract

Modeling open-vocabulary language fields in 3D is essential for intuitive human-AI interaction and querying within physical environments. State-of-the-art approaches, such as LangSplat, leverage 3D Gaussian Splatting to efficiently construct these language fields, encoding features distilled from high-dimensional models like CLIP. However, this efficiency is currently offset by the requirement to train a scene-specific language autoencoder for feature compression, introducing a costly, per-scene optimization bottleneck that hinders deployment scalability. In this work, we introduce Gen-LangSplat, that eliminates this requirement by replacing the scene-wise autoencoder with a generalized autoencoder, pre-trained extensively on the large-scale ScanNet dataset. This architectural shift enables the use of a fixed, compact latent space for language features across any new scene without any scene-specific training. By removing this dependency, our entire language field construction process achieves a efficiency boost while delivering querying performance comparable to, or exceeding, the original LangSplat method. To validate our design choice, we perform a thorough ablation study empirically determining the optimal latent embedding dimension and quantifying representational fidelity using Mean Squared Error and cosine similarity between the original and reprojected 512-dimensional CLIP embeddings. Our results demonstrate that generalized embeddings can efficiently and accurately support open-vocabulary querying in novel 3D scenes, paving the way for scalable, real-time interactive 3D AI applications.