Geometry-Consistent 4D Gaussian Splatting for Sparse-Input Dynamic View Synthesis

TL;DR

This paper introduces GC-4DGS, a framework that enhances 4D Gaussian Splatting with geometric consistency, enabling high-quality, real-time dynamic scene rendering from sparse views, suitable for AIoT applications.

Contribution

The paper proposes a novel geometric consistency infusion into 4D Gaussian Splatting, improving sparse-input dynamic view synthesis with a dynamic consistency check and depth regularization.

Findings

Outperforms RF-DeRF and original 4DGS in PSNR by 2.62dB and 1.58dB.

Achieves high-quality rendering from sparse views in real-time.

Effective on resource-constrained IoT edge devices.

Abstract

Gaussian Splatting has been considered as a novel way for view synthesis of dynamic scenes, which shows great potential in AIoT applications such as digital twins. However, recent dynamic Gaussian Splatting methods significantly degrade when only sparse input views are available, limiting their applicability in practice. The issue arises from the incoherent learning of 4D geometry as input views decrease. This paper presents GC-4DGS, a novel framework that infuses geometric consistency into 4D Gaussian Splatting (4DGS), offering real-time and high-quality dynamic scene rendering from sparse input views. While learning-based Multi-View Stereo (MVS) and monocular depth estimators (MDEs) provide geometry priors, directly integrating these with 4DGS yields suboptimal results due to the ill-posed nature of sparse-input 4D geometric optimization. To address these problems, we introduce a dynamic consistency checking strategy to reduce estimation uncertainties of MVS across spacetime. Furthermore, we propose a global-local depth regularization approach to distill spatiotemporal-consistent geometric information from monocular depths, thereby enhancing the coherent geometry and appearance learning within the 4D volume. Extensive experiments on the popular N3DV and Technicolor datasets validate the effectiveness of GC-4DGS in rendering quality without sacrificing efficiency. Notably, our method outperforms RF-DeRF, the latest dynamic radiance field tailored for sparse-input dynamic view synthesis, and the original 4DGS by 2.62dB and 1.58dB in PSNR, respectively, with seamless deployability on resource-constrained IoT edge devices.