Nebula: Enable City-Scale 3D Gaussian Splatting in Virtual Reality via Collaborative Rendering and Accelerated Stereo Rasterization

TL;DR

Nebula is a framework that enables city-scale 3D Gaussian splatting in virtual reality by collaborative rendering and accelerated stereo rasterization, significantly reducing bandwidth and improving motion-to-photon latency.

Contribution

Nebula introduces a scalable, cloud-assisted rendering framework with novel LoD search and stereo rasterization techniques for large-scale 3D Gaussian splatting in VR.

Findings

Reduces data communication by 1925% compared to streaming.

Achieves 2.7× speedup in motion-to-photon latency.

Enables high-fidelity VR content delivery at city scale.

Abstract

3D Gaussian splatting (3DGS) has drawn significant attention in the architectural community recently. However, current architectural designs often overlook the 3DGS scalability, making them fragile for extremely large-scale 3DGS. Meanwhile, the VR bandwidth requirement makes it impossible to deliver high-fidelity and smooth VR content from the cloud. We present Nebula, a coherent acceleration framework for large-scale 3DGS collaborative rendering. Instead of streaming videos, Nebula streams intermediate results after the LoD search, reducing 1925% data communication between the cloud and the client. To further enhance the motion-to-photon experience, we introduce a temporal-aware LoD search in the cloud that tames the irregular memory access and reduces redundant data access by exploiting temporal coherence across frames. On the client side, we propose a novel stereo rasterization that enables two eyes to share most computations during the stereo rendering with bit-accurate quality. With minimal hardware augmentations, Nebula achieves 2.7$\times$ motion-to-photon speedup and reduces 1925% bandwidth over lossy video streaming.