S2GS: Streaming Semantic Gaussian Splatting for Online Scene Understanding and Reconstruction

TL;DR

S2GS introduces a scalable, causal framework for online 3D scene understanding and reconstruction that updates incrementally without reprocessing past data, outperforming offline methods in long sequence processing.

Contribution

It presents S2GS, a novel incremental 3D Gaussian semantic field framework that enables scalable online scene understanding and reconstruction without future frame reliance.

Findings

Matches or exceeds offline baselines on joint reconstruction and understanding.

Processes over 1,000 frames with minimal increase in runtime and memory.

Outperforms offline methods in long-horizon scalability.

Abstract

Existing offline feed-forward methods for joint scene understanding and reconstruction on long image streams often repeatedly perform global computation over an ever-growing set of past observations, causing runtime and GPU memory to increase rapidly with sequence length and limiting scalability. We propose Streaming Semantic Gaussian Splatting (S2GS), a strictly causal, incremental 3D Gaussian semantic field framework: it does not leverage future frames and continuously updates scene geometry, appearance, and instance-level semantics without reprocessing historical frames, enabling scalable online joint reconstruction and understanding. S2GS adopts a geometry-semantic decoupled dual-backbone design: the geometry branch performs causal modeling to drive incremental Gaussian updates, while the semantic branch leverages a 2D foundation vision model and a query-driven decoder to predict segmentation masks and identity embeddings, further stabilized by query-level contrastive alignment and lightweight online association with an instance memory. Experiments show that S2GS matches or outperforms strong offline baselines on joint reconstruction-and-understanding benchmarks, while significantly improving long-horizon scalability: it processes 1,000+ frames with much slower growth in runtime and GPU memory, whereas offline global-processing baselines typically run out of memory at around 80 frames under the same setting.