Collaborative Regression of Expressive Bodies using Moderation

TL;DR

PIXIE is a novel method that reconstructs detailed, animatable 3D whole-body avatars with realistic facial features from a single image by intelligently merging expert estimates and considering gender-specific shape variations.

Contribution

Introduces a moderator that combines body, face, and hand features based on confidence, and incorporates gender-aware shape modeling for improved 3D human reconstruction.

Findings

Outperforms state-of-the-art in shape accuracy and facial detail.

Effectively merges expert features using confidence-weighted moderation.

Accurately estimates gender-specific 3D body shapes.

Abstract

Recovering expressive humans from images is essential for understanding human behavior. Methods that estimate 3D bodies, faces, or hands have progressed significantly, yet separately. Face methods recover accurate 3D shape and geometric details, but need a tight crop and struggle with extreme views and low resolution. Whole-body methods are robust to a wide range of poses and resolutions, but provide only a rough 3D face shape without details like wrinkles. To get the best of both worlds, we introduce PIXIE, which produces animatable, whole-body 3D avatars with realistic facial detail, from a single image. For this, PIXIE uses two key observations. First, existing work combines independent estimates from body, face, and hand experts, by trusting them equally. PIXIE introduces a novel moderator that merges the features of the experts, weighted by their confidence. All part experts can contribute to the whole, using SMPL-X's shared shape space across all body parts. Second, human shape is highly correlated with gender, but existing work ignores this. We label training images as male, female, or non-binary, and train PIXIE to infer "gendered" 3D body shapes with a novel shape loss. In addition to 3D body pose and shape parameters, PIXIE estimates expression, illumination, albedo and 3D facial surface displacements. Quantitative and qualitative evaluation shows that PIXIE estimates more accurate whole-body shape and detailed face shape than the state of the art. Models and code are available at https://pixie.is.tue.mpg.de.