Sparse-View 3D Gaussian Splatting in the Wild

TL;DR

This paper introduces a novel 3D sparse-view synthesis framework that effectively handles unconstrained real-world scenarios with distractors, using reference-guided refinement and Gaussian strategies to improve rendering quality.

Contribution

It presents a new method combining diffusion-based view refinement and Gaussian field strategies to enhance 3D rendering from sparse, unconstrained image collections.

Findings

Outperforms existing methods with significant PSNR, SSIM, and LPIPS improvements.

Achieves high-fidelity 3D rendering in unconstrained real-world scenarios.

Demonstrates robustness to distractors and transient elements.

Abstract

We propose a 3D novel sparse-view synthesis framework for unconstrained real-world scenarios that contain distractors. Unlike existing methods that primarily perform novel-view synthesis from a sparse set of constrained images without transient elements or leverage unconstrained dense image collections to enhance 3D representation in real-world scenarios, our method not only effectively tackles sparse unconstrained image collections, but also shows high-quality 3D rendering results. To do this, we introduce reference-guided view refinement with a diffusion model using a transient mask and a reference image to enhance the 3D representation and mitigate artifacts in rendered views. Furthermore, we address sparse regions in the Gaussian field via pseudo-view generation along with a sparsity-aware Gaussian replication strategy to amplify Gaussians in the sparse regions. Extensive experiments on publicly available datasets demonstrate that our methodology consistently outperforms existing methods (e.g., PSNR - 17.2%, SSIM - 10.8%, LPIPS - 4.0%) and provides high-fidelity 3D rendering results. This advancement paves the way for realizing unconstrained real-world scenarios without labor-intensive data acquisition. Our project page is available at $\href{https://robotic-vision-lab.github.io/SaveWildGS/}{here}$