Joint Material and Illumination Estimation from Photo Sets in the Wild

TL;DR

This paper introduces a method to jointly estimate material properties and illumination from sets of uncontrolled photos, enabling realistic editing and manipulation of images by leveraging multiple views and materials.

Contribution

It presents a novel optimization approach using neural networks trained on synthetic data to accurately recover non-diffuse materials and environment lighting from photo sets in the wild.

Findings

Achieves high-quality material and illumination estimates on real and synthetic images.

Outperforms state-of-the-art methods in qualitative user studies.

Enables photo-consistent image manipulation in uncontrolled settings.

Abstract

Faithful manipulation of shape, material, and illumination in 2D Internet images would greatly benefit from a reliable factorization of appearance into material (i.e., diffuse and specular) and illumination (i.e., environment maps). On the one hand, current methods that produce very high fidelity results, typically require controlled settings, expensive devices, or significant manual effort. To the other hand, methods that are automatic and work on 'in the wild' Internet images, often extract only low-frequency lighting or diffuse materials. In this work, we propose to make use of a set of photographs in order to jointly estimate the non-diffuse materials and sharp lighting in an uncontrolled setting. Our key observation is that seeing multiple instances of the same material under different illumination (i.e., environment), and different materials under the same illumination provide valuable constraints that can be exploited to yield a high-quality solution (i.e., specular materials and environment illumination) for all the observed materials and environments. Similar constraints also arise when observing multiple materials in a single environment, or a single material across multiple environments. The core of this approach is an optimization procedure that uses two neural networks that are trained on synthetic images to predict good gradients in parametric space given observation of reflected light. We evaluate our method on a range of synthetic and real examples to generate high-quality estimates, qualitatively compare our results against state-of-the-art alternatives via a user study, and demonstrate photo-consistent image manipulation that is otherwise very challenging to achieve.