Streaming Real-Time Rendered Scenes as 3D Gaussians

TL;DR

This paper proposes streaming a live 3D Gaussian scene representation instead of video to improve latency compensation and viewpoint flexibility in cloud rendering for gaming and XR.

Contribution

It introduces a Unity-based system that constructs and streams an optimized 3D Gaussian model from real-time views, enabling flexible remote rendering and dynamic scene updates.

Findings

The system supports relighting and rigid object dynamics.

It improves viewpoint flexibility over traditional video streaming.

The approach enables better amortization of scene modeling across users.

Abstract

Cloud rendering is widely used in gaming and XR to overcome limited client-side GPU resources and to support heterogeneous devices. Existing systems typically deliver the rendered scene as a 2D video stream, which tightly couples the transmitted content to the server-rendered viewpoint and limits latency compensation to image-space reprojection or warping. In this paper, we investigate an alternative approach based on streaming a live 3D Gaussian Splatting (3DGS) scene representation instead of only rendered video. We present a Unity-based prototype in which a server constructs and continuously optimizes a 3DGS model from real-time rendered reference views, while streaming the evolving representation to remote clients using full model snapshots and incremental updates supporting relighting and rigid object dynamics. The clients reconstruct the streamed Gaussian model locally and render their current viewpoint from the received representation. This approach aims to improve viewpoint flexibility for latency compensation and to better amortize server-side scene modeling across multiple users than per-user rendering and video streaming. We describe the system design, evaluate it, and compare it with conventional image warping.