MaskGaussian: Adaptive 3D Gaussian Representation from Probabilistic Masks

TL;DR

MaskGaussian introduces a probabilistic approach to 3D Gaussian representation, enabling dynamic assessment and pruning of Gaussians, which improves rendering quality while significantly reducing memory usage.

Contribution

It models Gaussians as probabilistic entities and proposes a masked-rasterization technique for dynamic Gaussian importance assessment.

Findings

Achieves better rendering quality with fewer Gaussians.

Prunes over 60% of Gaussians with minimal PSNR loss.

Demonstrates superior performance over previous pruning methods.

Abstract

While 3D Gaussian Splatting (3DGS) has demonstrated remarkable performance in novel view synthesis and real-time rendering, the high memory consumption due to the use of millions of Gaussians limits its practicality. To mitigate this issue, improvements have been made by pruning unnecessary Gaussians, either through a hand-crafted criterion or by using learned masks. However, these methods deterministically remove Gaussians based on a snapshot of the pruning moment, leading to sub-optimized reconstruction performance from a long-term perspective. To address this issue, we introduce MaskGaussian, which models Gaussians as probabilistic entities rather than permanently removing them, and utilize them according to their probability of existence. To achieve this, we propose a masked-rasterization technique that enables unused yet probabilistically existing Gaussians to receive gradients, allowing for dynamic assessment of their contribution to the evolving scene and adjustment of their probability of existence. Hence, the importance of Gaussians iteratively changes and the pruned Gaussians are selected diversely. Extensive experiments demonstrate the superiority of the proposed method in achieving better rendering quality with fewer Gaussians than previous pruning methods, pruning over 60% of Gaussians on average with only a 0.02 PSNR decline. Our code can be found at: https://github.com/kaikai23/MaskGaussian