StreamSplat: Towards Online Dynamic 3D Reconstruction from Uncalibrated Video Streams

TL;DR

StreamSplat introduces a fast, online framework for real-time 3D scene reconstruction from uncalibrated video streams, outperforming traditional optimization-based methods in speed and quality.

Contribution

It presents a novel feed-forward approach with probabilistic sampling, bidirectional deformation, and adaptive Gaussian fusion for online dynamic 3D reconstruction.

Findings

Achieves state-of-the-art quality on standard benchmarks.

Supports arbitrarily long video streams with 1200x speedup.

Outperforms optimization-based methods in speed and accuracy.

Abstract

Real-time reconstruction of dynamic 3D scenes from uncalibrated video streams demands robust online methods that recover scene dynamics from sparse observations under strict latency and memory constraints. Yet most dynamic reconstruction methods rely on hours of per-scene optimization under full-sequence access, limiting practical deployment. In this work, we introduce StreamSplat, a fully feed-forward framework that instantly transforms uncalibrated video streams of arbitrary length into dynamic 3D Gaussian Splatting (3DGS) representations in an online manner. It is achieved via three key technical innovations: 1) a probabilistic sampling mechanism that robustly predicts 3D Gaussians from uncalibrated inputs; 2) a bidirectional deformation field that yields reliable associations across frames and mitigates long-term error accumulation; 3) an adaptive Gaussian fusion operation that propagates persistent Gaussians while handling emerging and vanishing ones. Extensive experiments on standard dynamic and static benchmarks demonstrate that StreamSplat achieves state-of-the-art reconstruction quality and dynamic scene modeling. Uniquely, our method supports the online reconstruction of arbitrarily long video streams with a 1200x speedup over optimization-based methods. Our code and models are available at https://streamsplat3d.github.io/.