Dance In the Wild: Monocular Human Animation with Neural Dynamic Appearance Synthesis

TL;DR

This paper presents a novel neural dynamic appearance synthesis method for human animation in videos, effectively handling complex textures and motions, and achieving state-of-the-art results in in-the-wild scenarios.

Contribution

It introduces a StyleGAN-based architecture with a new motion signature for improved dynamic appearance synthesis and temporal coherence in human video animation.

Findings

Achieves high-quality in-the-wild human animation results

Outperforms previous methods quantitatively and qualitatively

Effectively handles loose garments and complex textures

Abstract

Synthesizing dynamic appearances of humans in motion plays a central role in applications such as AR/VR and video editing. While many recent methods have been proposed to tackle this problem, handling loose garments with complex textures and high dynamic motion still remains challenging. In this paper, we propose a video based appearance synthesis method that tackles such challenges and demonstrates high quality results for in-the-wild videos that have not been shown before. Specifically, we adopt a StyleGAN based architecture to the task of person specific video based motion retargeting. We introduce a novel motion signature that is used to modulate the generator weights to capture dynamic appearance changes as well as regularizing the single frame based pose estimates to improve temporal coherency. We evaluate our method on a set of challenging videos and show that our approach achieves state-of-the art performance both qualitatively and quantitatively.