Modeling Clothing as a Separate Layer for an Animatable Human Avatar

TL;DR

This paper introduces a two-layer mesh representation for animatable human avatars that explicitly models clothing separately from the body, enabling photorealistic and editable clothing animation from multi-view videos.

Contribution

It proposes a novel two-layer mesh and autoencoder-based approach for explicit clothing modeling, improving animation fidelity and enabling clothing texture editing.

Findings

Photorealistic clothing animation demonstrated for three actors.

Explicit clothing layer improves over single-layer avatars.

Clothing texture editing is enabled in animation output.

Abstract

We have recently seen great progress in building photorealistic animatable full-body codec avatars, but generating high-fidelity animation of clothing is still difficult. To address these difficulties, we propose a method to build an animatable clothed body avatar with an explicit representation of the clothing on the upper body from multi-view captured videos. We use a two-layer mesh representation to register each 3D scan separately with the body and clothing templates. In order to improve the photometric correspondence across different frames, texture alignment is then performed through inverse rendering of the clothing geometry and texture predicted by a variational autoencoder. We then train a new two-layer codec avatar with separate modeling of the upper clothing and the inner body layer. To learn the interaction between the body dynamics and clothing states, we use a temporal convolution network to predict the clothing latent code based on a sequence of input skeletal poses. We show photorealistic animation output for three different actors, and demonstrate the advantage of our clothed-body avatars over the single-layer avatars used in previous work. We also show the benefit of an explicit clothing model that allows the clothing texture to be edited in the animation output.