Sp2360: Sparse-view 360 Scene Reconstruction using Cascaded 2D Diffusion Priors

TL;DR

This paper introduces SparseSplat360, a method that leverages 2D diffusion priors and 3D Gaussian scene representations to effectively reconstruct 360-degree scenes from sparse views, outperforming existing methods.

Contribution

The paper presents a novel sparse-view 360 scene reconstruction approach using cascaded 2D diffusion models and explicit 3D Gaussian representations, enabling high-quality reconstructions from minimal views.

Findings

Outperforms existing sparse-view reconstruction methods on Mip-NeRF360 dataset.

Generates detailed 360 scenes from as few as 9 input views.

Uses explicit 3D Gaussian scene representation for faster training and rendering.

Abstract

We aim to tackle sparse-view reconstruction of a 360 3D scene using priors from latent diffusion models (LDM). The sparse-view setting is ill-posed and underconstrained, especially for scenes where the camera rotates 360 degrees around a point, as no visual information is available beyond some frontal views focused on the central object(s) of interest. In this work, we show that pretrained 2D diffusion models can strongly improve the reconstruction of a scene with low-cost fine-tuning. Specifically, we present SparseSplat360 (Sp2360), a method that employs a cascade of in-painting and artifact removal models to fill in missing details and clean novel views. Due to superior training and rendering speeds, we use an explicit scene representation in the form of 3D Gaussians over NeRF-based implicit representations. We propose an iterative update strategy to fuse generated pseudo novel views with existing 3D Gaussians fitted to the initial sparse inputs. As a result, we obtain a multi-view consistent scene representation with details coherent with the observed inputs. Our evaluation on the challenging Mip-NeRF360 dataset shows that our proposed 2D to 3D distillation algorithm considerably improves the performance of a regularized version of 3DGS adapted to a sparse-view setting and outperforms existing sparse-view reconstruction methods in 360 scene reconstruction. Qualitatively, our method generates entire 360 scenes from as few as 9 input views, with a high degree of foreground and background detail.