Relighting Humans: Occlusion-Aware Inverse Rendering for Full-Body Human Images

TL;DR

This paper presents a novel method for relighting human images by inferring occlusion-aware lighting and shape information directly from images, improving realism especially in hollowed regions.

Contribution

It introduces the first approach to directly infer light occlusion in the spherical harmonics framework using CNNs, even with limited training data.

Findings

Achieves more realistic relighting compared to occlusion-ignored methods.

Successfully infers plausible geometric and occlusion information from small datasets.

Enhances image synthesis applications with improved lighting inference.

Abstract

Relighting of human images has various applications in image synthesis. For relighting, we must infer albedo, shape, and illumination from a human portrait. Previous techniques rely on human faces for this inference, based on spherical harmonics (SH) lighting. However, because they often ignore light occlusion, inferred shapes are biased and relit images are unnaturally bright particularly at hollowed regions such as armpits, crotches, or garment wrinkles. This paper introduces the first attempt to infer light occlusion in the SH formulation directly. Based on supervised learning using convolutional neural networks (CNNs), we infer not only an albedo map, illumination but also a light transport map that encodes occlusion as nine SH coefficients per pixel. The main difficulty in this inference is the lack of training datasets compared to unlimited variations of human portraits. Surprisingly, geometric information including occlusion can be inferred plausibly even with a small dataset of synthesized human figures, by carefully preparing the dataset so that the CNNs can exploit the data coherency. Our method accomplishes more realistic relighting than the occlusion-ignored formulation.