CLM: Removing the GPU Memory Barrier for 3D Gaussian Splatting

TL;DR

This paper introduces CLM, a system that enables 3D Gaussian Splatting to render large scenes on a single consumer GPU by offloading data to CPU memory and optimizing data transfer and computation pipelines.

Contribution

CLM is the first system to efficiently scale 3D Gaussian Splatting to large scenes on a single consumer GPU through novel offloading and pipelining strategies.

Findings

Supports rendering of scenes with 100 million Gaussians on RTX4090

Achieves state-of-the-art reconstruction quality

Reduces communication overhead in GPU-CPU data transfer

Abstract

3D Gaussian Splatting (3DGS) is an increasingly popular novel view synthesis approach due to its fast rendering time, and high-quality output. However, scaling 3DGS to large (or intricate) scenes is challenging due to its large memory requirement, which exceed most GPU's memory capacity. In this paper, we describe CLM, a system that allows 3DGS to render large scenes using a single consumer-grade GPU, e.g., RTX4090. It does so by offloading Gaussians to CPU memory, and loading them into GPU memory only when necessary. To reduce performance and communication overheads, CLM uses a novel offloading strategy that exploits observations about 3DGS's memory access pattern for pipelining, and thus overlap GPU-to-CPU communication, GPU computation and CPU computation. Furthermore, we also exploit observation about the access pattern to reduce communication volume. Our evaluation shows that the resulting implementation can render a large scene that requires 100 million Gaussians on a single RTX4090 and achieve state-of-the-art reconstruction quality.