PERSONA: Personalized Whole-Body 3D Avatar with Pose-Driven Deformations from a Single Image

TL;DR

PERSONA creates personalized 3D human avatars with pose-driven deformations from a single image by combining diffusion-based video generation and 3D optimization, ensuring high-quality, pose-diverse renderings.

Contribution

The paper introduces PERSONA, a novel framework that integrates diffusion-based pose-rich video synthesis with 3D avatar optimization from a single image, overcoming previous limitations.

Findings

Effective pose-driven deformations from a single image.

High-quality, authentic renderings across diverse poses.

Mitigation of identity shifts through balanced sampling.

Abstract

Two major approaches exist for creating animatable human avatars. The first, a 3D-based approach, optimizes a NeRF- or 3DGS-based avatar from videos of a single person, achieving personalization through a disentangled identity representation. However, modeling pose-driven deformations, such as non-rigid cloth deformations, requires numerous pose-rich videos, which are costly and impractical to capture in daily life. The second, a diffusion-based approach, learns pose-driven deformations from large-scale in-the-wild videos but struggles with identity preservation and pose-dependent identity entanglement. We present PERSONA, a framework that combines the strengths of both approaches to obtain a personalized 3D human avatar with pose-driven deformations from a single image. PERSONA leverages a diffusion-based approach to generate pose-rich videos from the input image and optimizes a 3D avatar based on them. To ensure high authenticity and sharp renderings across diverse poses, we introduce balanced sampling and geometry-weighted optimization. Balanced sampling oversamples the input image to mitigate identity shifts in diffusion-generated training videos. Geometry-weighted optimization prioritizes geometry constraints over image loss, preserving rendering quality in diverse poses.