Gaussian-Flow: 4D Reconstruction with Dynamic 3D Gaussian Particle

TL;DR

Gaussian-Flow is a fast, point-based 4D scene reconstruction method that models complex scene deformations over time using a novel dual-domain deformation model, outperforming NeRF-based approaches in speed and quality.

Contribution

It introduces a dual-domain deformation model for Gaussian points, enabling efficient 4D scene reconstruction without separate training for each frame.

Findings

Achieves 5x faster training than per-frame 3DGS.

Outperforms previous methods in rendering quality.

Handles complex scene deformations over long videos.

Abstract

We introduce Gaussian-Flow, a novel point-based approach for fast dynamic scene reconstruction and real-time rendering from both multi-view and monocular videos. In contrast to the prevalent NeRF-based approaches hampered by slow training and rendering speeds, our approach harnesses recent advancements in point-based 3D Gaussian Splatting (3DGS). Specifically, a novel Dual-Domain Deformation Model (DDDM) is proposed to explicitly model attribute deformations of each Gaussian point, where the time-dependent residual of each attribute is captured by a polynomial fitting in the time domain, and a Fourier series fitting in the frequency domain. The proposed DDDM is capable of modeling complex scene deformations across long video footage, eliminating the need for training separate 3DGS for each frame or introducing an additional implicit neural field to model 3D dynamics. Moreover, the explicit deformation modeling for discretized Gaussian points ensures ultra-fast training and rendering of a 4D scene, which is comparable to the original 3DGS designed for static 3D reconstruction. Our proposed approach showcases a substantial efficiency improvement, achieving a $5\times$ faster training speed compared to the per-frame 3DGS modeling. In addition, quantitative results demonstrate that the proposed Gaussian-Flow significantly outperforms previous leading methods in novel view rendering quality. Project page: https://nju-3dv.github.io/projects/Gaussian-Flow