Learning Illumination from Diverse Portraits

TL;DR

This paper introduces a learning-based method to estimate HDR omnidirectional illumination from a single portrait image, enabling realistic virtual object integration in real-time on smartphones.

Contribution

The authors develop a novel training dataset and a deep learning model that outperforms existing methods in portrait lighting estimation, including handling diverse skin tones and real-time inference.

Findings

Outperforms state-of-the-art lighting estimation techniques.

Handles diverse skin tones and lighting conditions.

Enables real-time AR applications on smartphones.

Abstract

We present a learning-based technique for estimating high dynamic range (HDR), omnidirectional illumination from a single low dynamic range (LDR) portrait image captured under arbitrary indoor or outdoor lighting conditions. We train our model using portrait photos paired with their ground truth environmental illumination. We generate a rich set of such photos by using a light stage to record the reflectance field and alpha matte of 70 diverse subjects in various expressions. We then relight the subjects using image-based relighting with a database of one million HDR lighting environments, compositing the relit subjects onto paired high-resolution background imagery recorded during the lighting acquisition. We train the lighting estimation model using rendering-based loss functions and add a multi-scale adversarial loss to estimate plausible high frequency lighting detail. We show that our technique outperforms the state-of-the-art technique for portrait-based lighting estimation, and we also show that our method reliably handles the inherent ambiguity between overall lighting strength and surface albedo, recovering a similar scale of illumination for subjects with diverse skin tones. We demonstrate that our method allows virtual objects and digital characters to be added to a portrait photograph with consistent illumination. Our lighting inference runs in real-time on a smartphone, enabling realistic rendering and compositing of virtual objects into live video for augmented reality applications.