GSStream: 3D Gaussian Splatting based Volumetric Scene Streaming System

TL;DR

GSStream is a volumetric scene streaming system that leverages 3D Gaussian splatting, integrating viewport prediction and deep reinforcement learning for efficient, high-quality real-time scene delivery.

Contribution

The paper introduces GSStream, a novel system combining viewport prediction and DRL-based bitrate adaptation for efficient 3D Gaussian splatting scene streaming.

Findings

Outperforms existing systems in visual quality

Reduces network bandwidth usage

Supports real-time volumetric scene delivery

Abstract

Recently, the 3D Gaussian splatting (3DGS) technique for real-time radiance field rendering has revolutionized the field of volumetric scene representation, providing users with an immersive experience. But in return, it also poses a large amount of data volume, which is extremely bandwidth-intensive. Cutting-edge researchers have tried to introduce different approaches and construct multiple variants for 3DGS to obtain a more compact scene representation, but it is still challenging for real-time distribution. In this paper, we propose GSStream, a novel volumetric scene streaming system to support 3DGS data format. Specifically, GSStream integrates a collaborative viewport prediction module to better predict users' future behaviors by learning collaborative priors and historical priors from multiple users and users' viewport sequences and a deep reinforcement learning (DRL)-based bitrate adaptation module to tackle the state and action space variability challenge of the bitrate adaptation problem, achieving efficient volumetric scene delivery. Besides, we first build a user viewport trajectory dataset for volumetric scenes to support the training and streaming simulation. Extensive experiments prove that our proposed GSStream system outperforms existing representative volumetric scene streaming systems in visual quality and network usage. Demo video: https://youtu.be/3WEe8PN8yvA.