Relighting4D: Neural Relightable Human from Videos

TL;DR

Relighting4D is a novel neural framework that enables free-viewpoint relighting of humans from videos without expensive data, by decomposing geometry and reflectance into neural fields and applying physically based rendering.

Contribution

It introduces a self-supervised neural approach to decompose human geometry and reflectance from videos for relighting at arbitrary viewpoints, overcoming previous data and viewpoint limitations.

Findings

Capable of relighting dynamic humans with free viewpoints.

Works on both real and synthetic datasets.

Uses physically informed priors for regularization.

Abstract

Human relighting is a highly desirable yet challenging task. Existing works either require expensive one-light-at-a-time (OLAT) captured data using light stage or cannot freely change the viewpoints of the rendered body. In this work, we propose a principled framework, Relighting4D, that enables free-viewpoints relighting from only human videos under unknown illuminations. Our key insight is that the space-time varying geometry and reflectance of the human body can be decomposed as a set of neural fields of normal, occlusion, diffuse, and specular maps. These neural fields are further integrated into reflectance-aware physically based rendering, where each vertex in the neural field absorbs and reflects the light from the environment. The whole framework can be learned from videos in a self-supervised manner, with physically informed priors designed for regularization. Extensive experiments on both real and synthetic datasets demonstrate that our framework is capable of relighting dynamic human actors with free-viewpoints.