GaussianVLM: Scene-centric 3D Vision-Language Models using Language-aligned Gaussian Splats for Embodied Reasoning and Beyond

TL;DR

GaussianVLM introduces a scene-centric 3D vision-language model that embeds linguistic features into Gaussian splat scenes, enabling efficient, task-aware representations and significantly improving out-of-domain 3D scene understanding.

Contribution

It is the first Gaussian splatting-based 3D VLM that directly integrates language into dense scene representations for improved generalization.

Findings

Achieves fivefold performance improvement over prior 3D VLMs.

Operates effectively with photorealistic 3D representations from RGB images.

Demonstrates strong generalization in out-of-domain scenarios.

Abstract

As multimodal language models advance, their application to 3D scene understanding is a fast-growing frontier, driving the development of 3D Vision-Language Models (VLMs). Current methods show strong dependence on object detectors, introducing processing bottlenecks and limitations in taxonomic flexibility. To address these limitations, we propose a scene-centric 3D VLM for 3D Gaussian splat scenes that employs language- and task-aware scene representations. Our approach directly embeds rich linguistic features into the 3D scene representation by associating language with each Gaussian primitive, achieving early modality alignment. To process the resulting dense representations, we introduce a dual sparsifier that distills them into compact, task-relevant tokens via task-guided and location-guided pathways, producing sparse, task-aware global and local scene tokens. Notably, we present the first Gaussian splatting-based VLM, leveraging photorealistic 3D representations derived from standard RGB images, demonstrating strong generalization: it improves performance of prior 3D VLM five folds, in out-of-the-domain settings.