Drivable Avatar Clothing: Faithful Full-Body Telepresence with Dynamic Clothing Driven by Sparse RGB-D Input

TL;DR

This paper introduces a novel method for creating photorealistic full-body avatars with dynamic clothing driven by sparse RGB-D data, combining a new tracking algorithm with appearance remapping for high fidelity.

Contribution

The paper presents a Neural ICP algorithm for efficient garment shape tracking and a remapping technique for detailed appearance reconstruction in drivable avatars.

Findings

Effective garment shape tracking with sparse depth input.

High-fidelity clothing dynamics and appearance in avatars.

Generalization to new environments demonstrated.

Abstract

Clothing is an important part of human appearance but challenging to model in photorealistic avatars. In this work we present avatars with dynamically moving loose clothing that can be faithfully driven by sparse RGB-D inputs as well as body and face motion. We propose a Neural Iterative Closest Point (N-ICP) algorithm that can efficiently track the coarse garment shape given sparse depth input. Given the coarse tracking results, the input RGB-D images are then remapped to texel-aligned features, which are fed into the drivable avatar models to faithfully reconstruct appearance details. We evaluate our method against recent image-driven synthesis baselines, and conduct a comprehensive analysis of the N-ICP algorithm. We demonstrate that our method can generalize to a novel testing environment, while preserving the ability to produce high-fidelity and faithful clothing dynamics and appearance.