SemanticSplat: Feed-Forward 3D Scene Understanding with Language-Aware Gaussian Fields

TL;DR

SemanticSplat introduces a feed-forward 3D scene understanding approach that unifies geometry, appearance, and semantics using language-aware Gaussian fields, enabling accurate and holistic scene comprehension from sparse views.

Contribution

It is the first to unify 3D Gaussians with semantic attributes in a feed-forward framework for holistic scene understanding.

Findings

Effective in promptable and open-vocabulary segmentation tasks.

Outperforms existing methods in geometry reconstruction quality.

Achieves coherent multi-modal semantic scene understanding.

Abstract

Holistic 3D scene understanding, which jointly models geometry, appearance, and semantics, is crucial for applications like augmented reality and robotic interaction. Existing feed-forward 3D scene understanding methods (e.g., LSM) are limited to extracting language-based semantics from scenes, failing to achieve holistic scene comprehension. Additionally, they suffer from low-quality geometry reconstruction and noisy artifacts. In contrast, per-scene optimization methods rely on dense input views, which reduces practicality and increases complexity during deployment. In this paper, we propose SemanticSplat, a feed-forward semantic-aware 3D reconstruction method, which unifies 3D Gaussians with latent semantic attributes for joint geometry-appearance-semantics modeling. To predict the semantic anisotropic Gaussians, SemanticSplat fuses diverse feature fields (e.g., LSeg, SAM) with a cost volume representation that stores cross-view feature similarities, enhancing coherent and accurate scene comprehension. Leveraging a two-stage distillation framework, SemanticSplat reconstructs a holistic multi-modal semantic feature field from sparse-view images. Experiments demonstrate the effectiveness of our method for 3D scene understanding tasks like promptable and open-vocabulary segmentation. Video results are available at https://semanticsplat.github.io.