CoherentGS: Sparse Novel View Synthesis with Coherent 3D Gaussians

TL;DR

CoherentGS introduces a regularized, structured Gaussian representation with depth-based initialization to improve sparse-view 3D novel view synthesis, achieving better reconstruction quality and coherence over prior methods.

Contribution

The paper proposes a novel regularized optimization framework with depth-based initialization for sparse-view 3D synthesis using structured Gaussians, enhancing coherence and quality.

Findings

Significant improvement over state-of-the-art sparse-view NeRF methods.

Effective control of Gaussian positions improves reconstruction coherence.

Depth-based initialization enhances optimization stability and results.

Abstract

The field of 3D reconstruction from images has rapidly evolved in the past few years, first with the introduction of Neural Radiance Field (NeRF) and more recently with 3D Gaussian Splatting (3DGS). The latter provides a significant edge over NeRF in terms of the training and inference speed, as well as the reconstruction quality. Although 3DGS works well for dense input images, the unstructured point-cloud like representation quickly overfits to the more challenging setup of extremely sparse input images (e.g., 3 images), creating a representation that appears as a jumble of needles from novel views. To address this issue, we propose regularized optimization and depth-based initialization. Our key idea is to introduce a structured Gaussian representation that can be controlled in 2D image space. We then constraint the Gaussians, in particular their position, and prevent them from moving independently during optimization. Specifically, we introduce single and multiview constraints through an implicit convolutional decoder and a total variation loss, respectively. With the coherency introduced to the Gaussians, we further constrain the optimization through a flow-based loss function. To support our regularized optimization, we propose an approach to initialize the Gaussians using monocular depth estimates at each input view. We demonstrate significant improvements compared to the state-of-the-art sparse-view NeRF-based approaches on a variety of scenes.