End-to-End Rate-Distortion Optimized 3D Gaussian Representation

TL;DR

This paper introduces RDO-Gaussian, an end-to-end rate-distortion optimized approach for 3D Gaussian representations that significantly reduces storage and improves performance in 3D rendering tasks.

Contribution

It proposes a novel RDO framework with dynamic pruning and ECVQ, and models region-specific colors for better rate-distortion trade-offs in 3D Gaussian representations.

Findings

Reduces 3D Gaussian size over 40x

Outperforms existing methods in rate-distortion

Effective on real and synthetic scenes

Abstract

3D Gaussian Splatting (3DGS) has become an emerging technique with remarkable potential in 3D representation and image rendering. However, the substantial storage overhead of 3DGS significantly impedes its practical applications. In this work, we formulate the compact 3D Gaussian learning as an end-to-end Rate-Distortion Optimization (RDO) problem and propose RDO-Gaussian that can achieve flexible and continuous rate control. RDO-Gaussian addresses two main issues that exist in current schemes: 1) Different from prior endeavors that minimize the rate under the fixed distortion, we introduce dynamic pruning and entropy-constrained vector quantization (ECVQ) that optimize the rate and distortion at the same time. 2) Previous works treat the colors of each Gaussian equally, while we model the colors of different regions and materials with learnable numbers of parameters. We verify our method on both real and synthetic scenes, showcasing that RDO-Gaussian greatly reduces the size of 3D Gaussian over 40x, and surpasses existing methods in rate-distortion performance.