Relightable Neural Human Assets from Multi-view Gradient Illuminations

TL;DR

UltraStage is a comprehensive 3D human dataset with multi-view and multi-illumination data, enabling advanced human modeling and relighting research through neural representations and detailed surface information.

Contribution

The paper introduces UltraStage, a new dataset with multi-view and multi-illumination captures, and proposes neural human assets for high-fidelity relighting and view synthesis.

Findings

Neural human assets achieve high detail and realism.

Gradient illuminations improve surface normal and material recovery.

Neural assets enable realistic relighting from single images.

Abstract

Human modeling and relighting are two fundamental problems in computer vision and graphics, where high-quality datasets can largely facilitate related research. However, most existing human datasets only provide multi-view human images captured under the same illumination. Although valuable for modeling tasks, they are not readily used in relighting problems. To promote research in both fields, in this paper, we present UltraStage, a new 3D human dataset that contains more than 2,000 high-quality human assets captured under both multi-view and multi-illumination settings. Specifically, for each example, we provide 32 surrounding views illuminated with one white light and two gradient illuminations. In addition to regular multi-view images, gradient illuminations help recover detailed surface normal and spatially-varying material maps, enabling various relighting applications. Inspired by recent advances in neural representation, we further interpret each example into a neural human asset which allows novel view synthesis under arbitrary lighting conditions. We show our neural human assets can achieve extremely high capture performance and are capable of representing fine details such as facial wrinkles and cloth folds. We also validate UltraStage in single image relighting tasks, training neural networks with virtual relighted data from neural assets and demonstrating realistic rendering improvements over prior arts. UltraStage will be publicly available to the community to stimulate significant future developments in various human modeling and rendering tasks. The dataset is available at https://miaoing.github.io/RNHA.