Generative Multiview Relighting for 3D Reconstruction under Extreme Illumination Variation

TL;DR

This paper introduces a novel multiview relighting diffusion approach that enables high-fidelity 3D reconstruction of objects under extreme illumination variations, especially capturing view-dependent shiny appearances.

Contribution

It proposes a relighting and reconstruction pipeline that outperforms prior methods in handling extreme lighting changes and recovering view-dependent effects.

Findings

Outperforms existing techniques in high-fidelity appearance reconstruction

Effectively recovers view-dependent shiny appearances

Validated on synthetic and real datasets

Abstract

Reconstructing the geometry and appearance of objects from photographs taken in different environments is difficult as the illumination and therefore the object appearance vary across captured images. This is particularly challenging for more specular objects whose appearance strongly depends on the viewing direction. Some prior approaches model appearance variation across images using a per-image embedding vector, while others use physically-based rendering to recover the materials and per-image illumination. Such approaches fail at faithfully recovering view-dependent appearance given significant variation in input illumination and tend to produce mostly diffuse results. We present an approach that reconstructs objects from images taken under different illuminations by first relighting the images under a single reference illumination with a multiview relighting diffusion model and then reconstructing the object's geometry and appearance with a radiance field architecture that is robust to the small remaining inconsistencies among the relit images. We validate our proposed approach on both synthetic and real datasets and demonstrate that it greatly outperforms existing techniques at reconstructing high-fidelity appearance from images taken under extreme illumination variation. Moreover, our approach is particularly effective at recovering view-dependent "shiny" appearance which cannot be reconstructed by prior methods.