SegSplat: Feed-forward Gaussian Splatting and Open-Set Semantic Segmentation

TL;DR

SegSplat introduces a fast, feed-forward 3D reconstruction framework that integrates open-vocabulary semantic understanding by predicting semantic indices alongside geometric attributes, enabling real-time, semantically rich 3D scene generation.

Contribution

It presents a novel method combining rapid 3D Gaussian splatting with open-set semantic segmentation without per-scene optimization, advancing real-time semantic 3D scene understanding.

Findings

Achieves geometric fidelity comparable to state-of-the-art methods.

Enables robust open-set semantic segmentation in a single pass.

Operates without per-scene semantic optimization.

Abstract

We have introduced SegSplat, a novel framework designed to bridge the gap between rapid, feed-forward 3D reconstruction and rich, open-vocabulary semantic understanding. By constructing a compact semantic memory bank from multi-view 2D foundation model features and predicting discrete semantic indices alongside geometric and appearance attributes for each 3D Gaussian in a single pass, SegSplat efficiently imbues scenes with queryable semantics. Our experiments demonstrate that SegSplat achieves geometric fidelity comparable to state-of-the-art feed-forward 3D Gaussian Splatting methods while simultaneously enabling robust open-set semantic segmentation, crucially \textit{without} requiring any per-scene optimization for semantic feature integration. This work represents a significant step towards practical, on-the-fly generation of semantically aware 3D environments, vital for advancing robotic interaction, augmented reality, and other intelligent systems.