Dr. Splat: Directly Referring 3D Gaussian Splatting via Direct Language Embedding Registration

TL;DR

Dr. Splat introduces a new method for open-vocabulary 3D scene understanding that directly links language embeddings with 3D Gaussian representations, improving performance without scene-specific optimization.

Contribution

It presents a novel language feature registration technique for 3D Gaussian Splatting that bypasses rendering and employs Product Quantization for efficient, scalable scene understanding.

Findings

Outperforms existing methods in 3D semantic segmentation

Achieves superior results in 3D object localization

Demonstrates effective open-vocabulary 3D scene understanding

Abstract

We introduce Dr. Splat, a novel approach for open-vocabulary 3D scene understanding leveraging 3D Gaussian Splatting. Unlike existing language-embedded 3DGS methods, which rely on a rendering process, our method directly associates language-aligned CLIP embeddings with 3D Gaussians for holistic 3D scene understanding. The key of our method is a language feature registration technique where CLIP embeddings are assigned to the dominant Gaussians intersected by each pixel-ray. Moreover, we integrate Product Quantization (PQ) trained on general large-scale image data to compactly represent embeddings without per-scene optimization. Experiments demonstrate that our approach significantly outperforms existing approaches in 3D perception benchmarks, such as open-vocabulary 3D semantic segmentation, 3D object localization, and 3D object selection tasks. For video results, please visit : https://drsplat.github.io/