MERLiN: Single-Shot Material Estimation and Relighting for Photometric Stereo

TL;DR

MERLiN introduces an attention-based neural network that performs single-shot inverse rendering and relighting, enabling accurate surface normal and material estimation from a single image, thus reducing the need for complex data acquisition in photometric stereo.

Contribution

It presents a unified framework combining inverse rendering and relighting with a physically-based model trained on synthetic data, improving material and shape estimation from a single image.

Findings

High-quality shape and material estimation achieved on real images.

Relit images are physically accurate and useful for normal estimation.

Framework generalizes well from synthetic to real-world data.

Abstract

Photometric stereo typically demands intricate data acquisition setups involving multiple light sources to recover surface normals accurately. In this paper, we propose MERLiN, an attention-based hourglass network that integrates single image-based inverse rendering and relighting within a single unified framework. We evaluate the performance of photometric stereo methods using these relit images and demonstrate how they can circumvent the underlying challenge of complex data acquisition. Our physically-based model is trained on a large synthetic dataset containing complex shapes with spatially varying BRDF and is designed to handle indirect illumination effects to improve material reconstruction and relighting. Through extensive qualitative and quantitative evaluation, we demonstrate that the proposed framework generalizes well to real-world images, achieving high-quality shape, material estimation, and relighting. We assess these synthetically relit images over photometric stereo benchmark methods for their physical correctness and resulting normal estimation accuracy, paving the way towards single-shot photometric stereo through physically-based relighting. This work allows us to address the single image-based inverse rendering problem holistically, applying well to both synthetic and real data and taking a step towards mitigating the challenge of data acquisition in photometric stereo.