Sparse-to-Complete: From Sparse Image Captures to Complete 3D Scenes

TL;DR

S2C-3D is a novel framework that reconstructs complete 3D scenes from as few as six to eight images using a specialized diffusion model, view planning, and view-consistency sampling.

Contribution

The paper introduces a scene-specific diffusion model, a view planning scheme, and a view-consistency sampling process for high-fidelity sparse-view 3D reconstruction.

Findings

Outperforms state-of-the-art methods in sparse-view 3D reconstruction.

Produces artifact-free, complete 3D scenes from minimal input images.

Demonstrates robustness to artifacts and missing regions.

Abstract

We introduce S2C-3D, a novel sparse-view 3D reconstruction framework for high-fidelity and complete scene reconstruction from as few as six to eight images. Our framework features three components: a specialized diffusion model for scene-specific image restoration, a training-free view-consistency conditioned sampling process in the diffusion model for refined Gaussian optimization, and a camera trajectory planning scheme to ensure comprehensive scene coverage. The specialized diffusion model is developed by finetuning a pretrained architecture on the input views and their corresponding degraded counterparts. The adaptation to the scene distribution allows the model to repair Gaussian renderings while effectively eliminating domain gaps. Meanwhile, the trajectory planning scheme optimizes scene coverage by connecting each newly sampled camera to its two nearest neighbors. By iteratively constructing paths and retaining only those that significantly enhance visibility, the scheme establishes a trajectory that covers the entire scene. To address multi-view conflicts, the view-consistency conditioned sampling process quantifies the consistency between neighboring repaired images. This information is injected as a condition into the sampling process of the frozen diffusion model, facilitating the generation of view-consistent images without additional training. Consequently, our approach produces high-fidelity 3D Gaussians that are robust to artifacts. Experimental results demonstrate that S2C-3D outperforms state-of-the-art methods, constructing high-quality scenes that are free from missing regions, blurring, or other artifacts with very sparse inputs. The source code and data are available at https://gapszju.github.io/S2C-3D.