Animatable Virtual Humans: Learning pose-dependent human representations in UV space for interactive performance synthesis

TL;DR

This paper introduces a new method for realistic, real-time virtual human animation by learning pose-dependent appearance and geometry in UV space, leveraging SMPL models for efficiency.

Contribution

It presents a novel approach that learns pose-dependent differences from SMPL models to improve realism and efficiency in virtual human rendering.

Findings

Achieves high realism in real-time virtual human animation.

Effectively encodes pose-dependent appearance and geometry in UV space.

Leverages SMPL models to constrain learning and improve efficiency.

Abstract

We propose a novel representation of virtual humans for highly realistic real-time animation and rendering in 3D applications. We learn pose dependent appearance and geometry from highly accurate dynamic mesh sequences obtained from state-of-the-art multiview-video reconstruction. Learning pose-dependent appearance and geometry from mesh sequences poses significant challenges, as it requires the network to learn the intricate shape and articulated motion of a human body. However, statistical body models like SMPL provide valuable a-priori knowledge which we leverage in order to constrain the dimension of the search space enabling more efficient and targeted learning and define pose-dependency. Instead of directly learning absolute pose-dependent geometry, we learn the difference between the observed geometry and the fitted SMPL model. This allows us to encode both pose-dependent appearance and geometry in the consistent UV space of the SMPL model. This approach not only ensures a high level of realism but also facilitates streamlined processing and rendering of virtual humans in real-time scenarios.