Consistency Diffusion Models for Single-Image 3D Reconstruction with Priors

TL;DR

This paper introduces a novel Consistency Diffusion Model for single-image 3D reconstruction that effectively integrates 2D and 3D priors within a Bayesian framework to improve reconstruction accuracy and consistency.

Contribution

It presents a pioneering diffusion-based training framework that incorporates 3D structural priors and 2D image priors, enhancing reconstruction consistency without shifting model learning.

Findings

Sets new benchmarks on synthetic datasets

Achieves superior results on real-world datasets

Improves reconstruction consistency and accuracy

Abstract

This paper delves into the study of 3D point cloud reconstruction from a single image. Our objective is to develop the Consistency Diffusion Model, exploring synergistic 2D and 3D priors in the Bayesian framework to ensure superior consistency in the reconstruction process, a challenging yet critical requirement in this field. Specifically, we introduce a pioneering training framework under diffusion models that brings two key innovations. First, we convert 3D structural priors derived from the initial 3D point cloud as a bound term to increase evidence in the variational Bayesian framework, leveraging these robust intrinsic priors to tightly govern the diffusion training process and bolster consistency in reconstruction. Second, we extract and incorporate 2D priors from the single input image, projecting them onto the 3D point cloud to enrich the guidance for diffusion training. Our framework not only sidesteps potential model learning shifts that may arise from directly imposing additional constraints during training but also precisely transposes the 2D priors into the 3D domain. Extensive experimental evaluations reveal that our approach sets new benchmarks in both synthetic and real-world datasets. The code is included with the submission.