How to Use Diffusion Priors under Sparse Views?

TL;DR

This paper introduces IPSM, a novel method leveraging visual inline priors from pose relationships to improve diffusion-based sparse-view 3D reconstruction without fine-tuning, achieving state-of-the-art results.

Contribution

The paper proposes IPSM and IPSM-Gaussian, new techniques that utilize pose-based inline priors to enhance diffusion priors in sparse-view 3D reconstruction without pre-training.

Findings

Achieves state-of-the-art reconstruction quality on public datasets.

Effectively rectifies diffusion guidance using pose-based inline priors.

Improves 3D reconstruction without fine-tuning or pre-training.

Abstract

Novel view synthesis under sparse views has been a long-term important challenge in 3D reconstruction. Existing works mainly rely on introducing external semantic or depth priors to supervise the optimization of 3D representations. However, the diffusion model, as an external prior that can directly provide visual supervision, has always underperformed in sparse-view 3D reconstruction using Score Distillation Sampling (SDS) due to the low information entropy of sparse views compared to text, leading to optimization challenges caused by mode deviation. To this end, we present a thorough analysis of SDS from the mode-seeking perspective and propose Inline Prior Guided Score Matching (IPSM), which leverages visual inline priors provided by pose relationships between viewpoints to rectify the rendered image distribution and decomposes the original optimization objective of SDS, thereby offering effective diffusion visual guidance without any fine-tuning or pre-training. Furthermore, we propose the IPSM-Gaussian pipeline, which adopts 3D Gaussian Splatting as the backbone and supplements depth and geometry consistency regularization based on IPSM to further improve inline priors and rectified distribution. Experimental results on different public datasets show that our method achieves state-of-the-art reconstruction quality. The code is released at https://github.com/iCVTEAM/IPSM.