HAD: Hallucination-Aware Diffusion Priors for 3D Reconstruction

TL;DR

HAD introduces a hallucination-aware diffusion prior that estimates pixel-wise hallucination scores to improve 3D reconstruction quality by reducing artifacts, leveraging multi-view reasoning and view-conditioned image fusion.

Contribution

The paper proposes a novel hallucination-aware diffusion prior that estimates pixel-wise hallucination scores and fuses multiple augmented views to enhance 3D reconstruction accuracy.

Findings

Reduces hallucination artifacts in diffusion-assisted 3D reconstruction.

Achieves state-of-the-art performance on multiple benchmarks.

Effectively fuses multiple views for improved novel view synthesis.

Abstract

Diffusion priors have recently demonstrated strong capability in enhancing the quality of sparse-view 3D reconstruction by augmenting training views at novel viewpoints, but they inevitably introduce hallucinated content -- artifacts inconsistent with the input views -- into the final 3D model. To address this challenge, we propose Hallucination-Aware Diffusion prior (HAD), which estimates pixel-wise hallucination score maps for augmented images by leveraging multi-view reasoning capabilities from a feedforward novel view synthesis (NVS) network pre-trained on large-scale 3D data. These hallucination scores enable selective masking of unreliable pixels during the progressive 3D reconstruction procedure, preventing the introduction of non-existent artifacts into the 3D model. To further enhance performance, we create multiple versions of augmented images at each novel view by conditioning the diffusion prior on different input views, which are then fused into a final image that leverages the broader context across all input views. We show that our method substantially reduces hallucination artifacts in diffusion-assisted 3D reconstruction, thereby achieving state-of-the-art performance across multiple benchmarks on novel view synthesis. Our project are publicly available at \href{https://xiliu8006.github.io/HAD-Project-website/}{project website}.