Creating and Reenacting Controllable 3D Humans with Differentiable Rendering

TL;DR

This paper introduces a neural rendering system that creates controllable 3D human models capable of appearance transfer and reenactment, leveraging a graph convolutional network and differentiable rendering for high-quality, artifact-free video synthesis.

Contribution

It presents a novel end-to-end architecture combining GCNs and differentiable rendering to reconstruct fully controllable 3D textured human models from videos.

Findings

Achieves high SSIM, LPIPS, MSE, and FVD scores in experiments.

Successfully models body deformations and preserves texture quality.

Enables control over pose and rendering parameters for human synthesis.

Abstract

This paper proposes a new end-to-end neural rendering architecture to transfer appearance and reenact human actors. Our method leverages a carefully designed graph convolutional network (GCN) to model the human body manifold structure, jointly with differentiable rendering, to synthesize new videos of people in different contexts from where they were initially recorded. Unlike recent appearance transferring methods, our approach can reconstruct a fully controllable 3D texture-mapped model of a person, while taking into account the manifold structure from body shape and texture appearance in the view synthesis. Specifically, our approach models mesh deformations with a three-stage GCN trained in a self-supervised manner on rendered silhouettes of the human body. It also infers texture appearance with a convolutional network in the texture domain, which is trained in an adversarial regime to reconstruct human texture from rendered images of actors in different poses. Experiments on different videos show that our method successfully infers specific body deformations and avoid creating texture artifacts while achieving the best values for appearance in terms of Structural Similarity (SSIM), Learned Perceptual Image Patch Similarity (LPIPS), Mean Squared Error (MSE), and Fr\'echet Video Distance (FVD). By taking advantages of both differentiable rendering and the 3D parametric model, our method is fully controllable, which allows controlling the human synthesis from both pose and rendering parameters. The source code is available at https://www.verlab.dcc.ufmg.br/retargeting-motion/wacv2022.