All-frequency Full-body Human Image Relighting

TL;DR

This paper introduces a two-stage neural relighting method for human images that accurately reproduces high-frequency shadows and shading by approximating environment lighting with area lights, improving physical realism.

Contribution

It presents a novel two-stage approach combining inverse rendering and differentiable shadow mapping to achieve physically-based high-frequency relighting of human images.

Findings

Successfully reproduces high-frequency shadows and shading

Outperforms existing methods in physical realism

Demonstrates plausible relighting results

Abstract

Relighting of human images enables post-photography editing of lighting effects in portraits. The current mainstream approach uses neural networks to approximate lighting effects without explicitly accounting for the principle of physical shading. As a result, it often has difficulty representing high-frequency shadows and shading. In this paper, we propose a two-stage relighting method that can reproduce physically-based shadows and shading from low to high frequencies. The key idea is to approximate an environment light source with a set of a fixed number of area light sources. The first stage employs supervised inverse rendering from a single image using neural networks and calculates physically-based shading. The second stage then calculates shadow for each area light and sums up to render the final image. We propose to make soft shadow mapping differentiable for the area-light approximation of environment lighting. We demonstrate that our method can plausibly reproduce all-frequency shadows and shading caused by environment illumination, which have been difficult to reproduce using existing methods.