Intrinsic Image Fusion for Multi-View 3D Material Reconstruction

TL;DR

This paper presents a novel intrinsic image fusion method that reconstructs high-quality 3D materials from multi-view images by integrating priors, diffusion-based estimations, and robust optimization to outperform existing techniques.

Contribution

It introduces a new fusion framework combining diffusion-based estimations and confidence-based optimization for improved multi-view 3D material reconstruction.

Findings

Outperforms state-of-the-art in material disentanglement

Produces sharp, clean reconstructions suitable for relighting

Effective in both synthetic and real scenes

Abstract

We introduce Intrinsic Image Fusion, a method that reconstructs high-quality physically based materials from multi-view images. Material reconstruction is highly underconstrained and typically relies on analysis-by-synthesis, which requires expensive and noisy path tracing. To better constrain the optimization, we incorporate single-view priors into the reconstruction process. We leverage a diffusion-based material estimator that produces multiple, but often inconsistent, candidate decompositions per view. To reduce the inconsistency, we fit an explicit low-dimensional parametric function to the predictions. We then propose a robust optimization framework using soft per-view prediction selection together with confidence-based soft multi-view inlier set to fuse the most consistent predictions of the most confident views into a consistent parametric material space. Finally, we use inverse path tracing to optimize for the low-dimensional parameters. Our results outperform state-of-the-art methods in material disentanglement on both synthetic and real scenes, producing sharp and clean reconstructions suitable for high-quality relighting.