MEGA: Memory-Efficient 4D Gaussian Splatting for Dynamic Scenes

TL;DR

This paper presents MEGA, a memory-efficient 4D Gaussian Splatting framework that significantly reduces memory usage while maintaining high-quality dynamic scene rendering, enabling practical large-scale 3D scene capture.

Contribution

It introduces a novel memory-efficient 4D Gaussian representation and entropy-based deformation technique, reducing storage costs by over 190 times compared to prior methods.

Findings

Achieves approximately 190× and 125× storage reduction on two datasets.

Maintains comparable rendering speed and scene quality.

Sets a new standard in memory-efficient dynamic scene representation.

Abstract

4D Gaussian Splatting (4DGS) has recently emerged as a promising technique for capturing complex dynamic 3D scenes with high fidelity. It utilizes a 4D Gaussian representation and a GPU-friendly rasterizer, enabling rapid rendering speeds. Despite its advantages, 4DGS faces significant challenges, notably the requirement of millions of 4D Gaussians, each with extensive associated attributes, leading to substantial memory and storage cost. This paper introduces a memory-efficient framework for 4DGS. We streamline the color attribute by decomposing it into a per-Gaussian direct color component with only 3 parameters and a shared lightweight alternating current color predictor. This approach eliminates the need for spherical harmonics coefficients, which typically involve up to 144 parameters in classic 4DGS, thereby creating a memory-efficient 4D Gaussian representation. Furthermore, we introduce an entropy-constrained Gaussian deformation technique that uses a deformation field to expand the action range of each Gaussian and integrates an opacity-based entropy loss to limit the number of Gaussians, thus forcing our model to use as few Gaussians as possible to fit a dynamic scene well. With simple half-precision storage and zip compression, our framework achieves a storage reduction by approximately 190$\times$ and 125$\times$ on the Technicolor and Neural 3D Video datasets, respectively, compared to the original 4DGS. Meanwhile, it maintains comparable rendering speeds and scene representation quality, setting a new standard in the field. Code is available at https://github.com/Xinjie-Q/MEGA.