Gaussians on a Diet: High-Quality Memory-Bounded 3D Gaussian Splatting Training

TL;DR

This paper introduces a memory-efficient training framework for 3D Gaussian Splatting that maintains high rendering quality while significantly reducing peak memory usage, enabling deployment on edge devices.

Contribution

A systematic memory-bounded training method that dynamically prunes and grows Gaussians, maintaining low memory footprint during high-quality 3D rendering training.

Findings

Achieves up to 80% lower peak memory consumption on NVIDIA Jetson AGX Xavier.

Maintains comparable visual quality to original 3DGS under memory constraints.

Demonstrates effectiveness across various real-world datasets.

Abstract

3D Gaussian Splatting (3DGS) has revolutionized novel view synthesis with high-quality rendering through continuous aggregations of millions of 3D Gaussian primitives. However, it suffers from a substantial memory footprint, particularly during training due to uncontrolled densification, posing a critical bottleneck for deployment on memory-constrained edge devices. While existing methods prune redundant Gaussians post-training, they fail to address the peak memory spikes caused by the abrupt growth of Gaussians early in the training process. To solve the training memory consumption problem, we propose a systematic memory-bounded training framework that dynamically optimizes Gaussians through iterative growth and pruning. In other words, the proposed framework alternates between incremental pruning of low-impact Gaussians and strategic growing of new primitives with an adaptive Gaussian compensation, maintaining a near-constant low memory usage while progressively refining rendering fidelity. We comprehensively evaluate the proposed training framework on various real-world datasets under strict memory constraints, showing significant improvements over existing state-of-the-art methods. Particularly, our proposed method practically enables memory-efficient 3DGS training on NVIDIA Jetson AGX Xavier, achieving similar visual quality with up to 80% lower peak training memory consumption than the original 3DGS.