Sparse4DGS: 4D Gaussian Splatting for Sparse-Frame Dynamic Scene Reconstruction

TL;DR

Sparse4DGS introduces a novel approach for 4D scene reconstruction from sparse frames, focusing on texture-rich areas to improve performance where dense data is unavailable.

Contribution

It is the first method to enable sparse-frame dynamic scene reconstruction, employing texture-aware regularization and optimization techniques.

Findings

Outperforms existing methods on multiple datasets with sparse inputs.

Effective in handling high-texture regions in sparse data scenarios.

Demonstrates robustness in real-world sparse-frame reconstructions.

Abstract

Dynamic Gaussian Splatting approaches have achieved remarkable performance for 4D scene reconstruction. However, these approaches rely on dense-frame video sequences for photorealistic reconstruction. In real-world scenarios, due to equipment constraints, sometimes only sparse frames are accessible. In this paper, we propose Sparse4DGS, the first method for sparse-frame dynamic scene reconstruction. We observe that dynamic reconstruction methods fail in both canonical and deformed spaces under sparse-frame settings, especially in areas with high texture richness. Sparse4DGS tackles this challenge by focusing on texture-rich areas. For the deformation network, we propose Texture-Aware Deformation Regularization, which introduces a texture-based depth alignment loss to regulate Gaussian deformation. For the canonical Gaussian field, we introduce Texture-Aware Canonical Optimization, which incorporates texture-based noise into the gradient descent process of canonical Gaussians. Extensive experiments show that when taking sparse frames as inputs, our method outperforms existing dynamic or few-shot techniques on NeRF-Synthetic, HyperNeRF, NeRF-DS, and our iPhone-4D datasets.