Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control

TL;DR

Neural Actor is a novel neural rendering approach that synthesizes high-quality, controllable human images from arbitrary viewpoints and poses, leveraging a canonical space and pose-dependent deformations.

Contribution

It introduces a neural implicit method using a body model proxy and residual deformations for photo-realistic human synthesis with pose and shape control.

Findings

Outperforms state-of-the-art in view synthesis quality

Generalizes well to unseen poses

Supports body shape manipulation

Abstract

We propose Neural Actor (NA), a new method for high-quality synthesis of humans from arbitrary viewpoints and under arbitrary controllable poses. Our method is built upon recent neural scene representation and rendering works which learn representations of geometry and appearance from only 2D images. While existing works demonstrated compelling rendering of static scenes and playback of dynamic scenes, photo-realistic reconstruction and rendering of humans with neural implicit methods, in particular under user-controlled novel poses, is still difficult. To address this problem, we utilize a coarse body model as the proxy to unwarp the surrounding 3D space into a canonical pose. A neural radiance field learns pose-dependent geometric deformations and pose- and view-dependent appearance effects in the canonical space from multi-view video input. To synthesize novel views of high fidelity dynamic geometry and appearance, we leverage 2D texture maps defined on the body model as latent variables for predicting residual deformations and the dynamic appearance. Experiments demonstrate that our method achieves better quality than the state-of-the-arts on playback as well as novel pose synthesis, and can even generalize well to new poses that starkly differ from the training poses. Furthermore, our method also supports body shape control of the synthesized results.