Structured 3D Features for Reconstructing Controllable Avatars

TL;DR

This paper presents Structured 3D Features, a novel implicit 3D representation that enables controllable, animatable 3D human avatars from a single image, surpassing previous methods in reconstruction, relighting, and editing capabilities.

Contribution

Introduction of Structured 3D Features, a new implicit 3D model that captures detailed human shape and appearance for controllable avatar reconstruction from minimal input.

Findings

Outperforms previous state-of-the-art in monocular 3D reconstruction.

Enables novel view synthesis, relighting, and re-posing.

Supports editing for virtual try-on applications.

Abstract

We introduce Structured 3D Features, a model based on a novel implicit 3D representation that pools pixel-aligned image features onto dense 3D points sampled from a parametric, statistical human mesh surface. The 3D points have associated semantics and can move freely in 3D space. This allows for optimal coverage of the person of interest, beyond just the body shape, which in turn, additionally helps modeling accessories, hair, and loose clothing. Owing to this, we present a complete 3D transformer-based attention framework which, given a single image of a person in an unconstrained pose, generates an animatable 3D reconstruction with albedo and illumination decomposition, as a result of a single end-to-end model, trained semi-supervised, and with no additional postprocessing. We show that our S3F model surpasses the previous state-of-the-art on various tasks, including monocular 3D reconstruction, as well as albedo and shading estimation. Moreover, we show that the proposed methodology allows novel view synthesis, relighting, and re-posing the reconstruction, and can naturally be extended to handle multiple input images (e.g. different views of a person, or the same view, in different poses, in video). Finally, we demonstrate the editing capabilities of our model for 3D virtual try-on applications.