S2D: Sparse to Dense Lifting for 3D Reconstruction with Minimal Inputs

TL;DR

S2D introduces a novel pipeline that effectively converts sparse inputs into high-quality dense 3D reconstructions, improving consistency and reducing input requirements for 3D scene generation.

Contribution

The paper presents a new two-fold lifting approach combining diffusion models and robust reconstruction strategies to enhance 3D Gaussian Splatting from minimal inputs.

Findings

Achieves superior consistency in novel view generation.

Provides high-fidelity reconstructions from sparse inputs.

Reduces the number of captures needed for 3D scene reconstruction.

Abstract

Explicit 3D representations have already become an essential medium for 3D simulation and understanding. However, the most commonly used point cloud and 3D Gaussian Splatting (3DGS) each suffer from non-photorealistic rendering and significant degradation under sparse inputs. In this paper, we introduce Sparse to Dense lifting (S2D), a novel pipeline that bridges the two representations and achieves high-quality 3DGS reconstruction with minimal inputs. Specifically, the S2D lifting is two-fold. We first present an efficient one-step diffusion model that lifts sparse point cloud for high-fidelity image artifact fixing. Meanwhile, to reconstruct 3D consistent scenes, we also design a corresponding reconstruction strategy with random sample drop and weighted gradient for robust model fitting from sparse input views to dense novel views. Extensive experiments show that S2D achieves the best consistency in generating novel view guidance and first-tier sparse view reconstruction quality under different input sparsity. By reconstructing stable scenes with the least possible captures among existing methods, S2D enables minimal input requirements for 3DGS applications.