SwinGS: Sliding Window Gaussian Splatting for Volumetric Video Streaming with Arbitrary Length

TL;DR

SwinGS is a real-time volumetric video streaming framework that uses a sliding window Gaussian approach combined with MCMC to adapt to dynamic scenes, significantly reducing transmission costs and supporting arbitrary video lengths.

Contribution

This paper introduces SwinGS, a novel method that enhances volumetric video streaming by enabling real-time, scalable, and efficient transmission of dynamic 3D scenes.

Findings

Reduces transmission costs by 83.6% compared to previous methods.

Supports volumetric videos of arbitrary length without increasing GPU resources.

Demonstrates real-time playback on various devices via WebGL viewer.

Abstract

Recent advances in 3D Gaussian Splatting (3DGS) have garnered significant attention in computer vision and computer graphics due to its high rendering speed and remarkable quality. While extant research has endeavored to extend the application of 3DGS from static to dynamic scenes, such efforts have been consistently impeded by excessive model sizes, constraints on video duration, and content deviation. These limitations significantly compromise the streamability of dynamic 3D Gaussian models, thereby restricting their utility in downstream applications, including volumetric video, autonomous vehicle, and immersive technologies such as virtual, augmented, and mixed reality. This paper introduces SwinGS, a novel framework for training, delivering, and rendering volumetric video in a real-time streaming fashion. To address the aforementioned challenges and enhance streamability, SwinGS integrates spacetime Gaussian with Markov Chain Monte Carlo (MCMC) to adapt the model to fit various 3D scenes across frames, in the meantime employing a sliding window captures Gaussian snapshots for each frame in an accumulative way. We implement a prototype of SwinGS and demonstrate its streamability across various datasets and scenes. Additionally, we develop an interactive WebGL viewer enabling real-time volumetric video playback on most devices with modern browsers, including smartphones and tablets. Experimental results show that SwinGS reduces transmission costs by 83.6% compared to previous work and could be easily scaled to volumetric videos with arbitrary length with no increasing of required GPU resources.