Relighting Humans in the Wild: Monocular Full-Body Human Relighting with Domain Adaptation

TL;DR

This paper introduces a two-stage neural network approach for monocular full-body human relighting that improves generalization across diverse materials and reduces domain gap, incorporating a deep video prior for dynamic illumination consistency.

Contribution

It presents a novel two-stage domain adaptation method for human relighting from a single image, enhancing non-diffuse reflection modeling and video consistency.

Findings

Improved relighting quality across various cloth textures.

Reduced synthetic-to-real domain gap.

Effective handling of dynamic illumination in videos.

Abstract

The modern supervised approaches for human image relighting rely on training data generated from 3D human models. However, such datasets are often small (e.g., Light Stage data with a small number of individuals) or limited to diffuse materials (e.g., commercial 3D scanned human models). Thus, the human relighting techniques suffer from the poor generalization capability and synthetic-to-real domain gap. In this paper, we propose a two-stage method for single-image human relighting with domain adaptation. In the first stage, we train a neural network for diffuse-only relighting. In the second stage, we train another network for enhancing non-diffuse reflection by learning residuals between real photos and images reconstructed by the diffuse-only network. Thanks to the second stage, we can achieve higher generalization capability against various cloth textures, while reducing the domain gap. Furthermore, to handle input videos, we integrate illumination-aware deep video prior to greatly reduce flickering artifacts even with challenging settings under dynamic illuminations.