Deceptive-NeRF/3DGS: Diffusion-Generated Pseudo-Observations for High-Quality Sparse-View Reconstruction

TL;DR

This paper introduces Deceptive-NeRF/3DGS, a method that uses diffusion-generated pseudo-observations to significantly improve high-quality 3D reconstruction from sparse input views, outperforming existing methods.

Contribution

It proposes a novel approach that directly trains NeRF/3DGS with diffusion-generated images as pseudo-observations, enhancing sparse-view reconstruction with limited input data.

Findings

Outperforms state-of-the-art methods on diverse datasets

Densifies sparse observations by 5 to 10 times

Enables super-resolution novel view synthesis from few views

Abstract

Novel view synthesis via Neural Radiance Fields (NeRFs) or 3D Gaussian Splatting (3DGS) typically necessitates dense observations with hundreds of input images to circumvent artifacts. We introduce Deceptive-NeRF/3DGS to enhance sparse-view reconstruction with only a limited set of input images, by leveraging a diffusion model pre-trained from multiview datasets. Different from using diffusion priors to regularize representation optimization, our method directly uses diffusion-generated images to train NeRF/3DGS as if they were real input views. Specifically, we propose a deceptive diffusion model turning noisy images rendered from few-view reconstructions into high-quality photorealistic pseudo-observations. To resolve consistency among pseudo-observations and real input views, we develop an uncertainty measure to guide the diffusion model's generation. Our system progressively incorporates diffusion-generated pseudo-observations into the training image sets, ultimately densifying the sparse input observations by 5 to 10 times. Extensive experiments across diverse and challenging datasets validate that our approach outperforms existing state-of-the-art methods and is capable of synthesizing novel views with super-resolution in the few-view setting.