Drivable Volumetric Avatars using Texel-Aligned Features

TL;DR

This paper introduces a novel volumetric avatar framework with texel-aligned features and articulated primitives, enabling photorealistic, dynamic, and view-generalizable full-body avatar synthesis without high-quality mesh tracking.

Contribution

It presents a new volumetric and texel-aligned feature-based approach that improves avatar realism and generalization over existing methods, especially in challenging poses and views.

Findings

Outperforms state-of-the-art in challenging driving scenarios

Supports detailed clothing dynamics like wrinkles

Does not require high-quality mesh tracking

Abstract

Photorealistic telepresence requires both high-fidelity body modeling and faithful driving to enable dynamically synthesized appearance that is indistinguishable from reality. In this work, we propose an end-to-end framework that addresses two core challenges in modeling and driving full-body avatars of real people. One challenge is driving an avatar while staying faithful to details and dynamics that cannot be captured by a global low-dimensional parameterization such as body pose. Our approach supports driving of clothed avatars with wrinkles and motion that a real driving performer exhibits beyond the training corpus. Unlike existing global state representations or non-parametric screen-space approaches, we introduce texel-aligned features -- a localised representation which can leverage both the structural prior of a skeleton-based parametric model and observed sparse image signals at the same time. Another challenge is modeling a temporally coherent clothed avatar, which typically requires precise surface tracking. To circumvent this, we propose a novel volumetric avatar representation by extending mixtures of volumetric primitives to articulated objects. By explicitly incorporating articulation, our approach naturally generalizes to unseen poses. We also introduce a localized viewpoint conditioning, which leads to a large improvement in generalization of view-dependent appearance. The proposed volumetric representation does not require high-quality mesh tracking as a prerequisite and brings significant quality improvements compared to mesh-based counterparts. In our experiments, we carefully examine our design choices and demonstrate the efficacy of our approach, outperforming the state-of-the-art methods on challenging driving scenarios.