Stratified Avatar Generation from Sparse Observations

TL;DR

This paper introduces a stratified approach for full-body avatar reconstruction from sparse AR/VR observations, decoupling upper and lower body reconstruction stages, and leveraging diffusion models for improved accuracy.

Contribution

It proposes a novel two-stage decoupled reconstruction method using diffusion models, enhancing full-body avatar estimation from limited input data.

Findings

State-of-the-art performance on AMASS dataset

Effective decoupling of upper and lower body reconstruction

Utilization of diffusion models improves motion generation

Abstract

Estimating 3D full-body avatars from AR/VR devices is essential for creating immersive experiences in AR/VR applications. This task is challenging due to the limited input from Head Mounted Devices, which capture only sparse observations from the head and hands. Predicting the full-body avatars, particularly the lower body, from these sparse observations presents significant difficulties. In this paper, we are inspired by the inherent property of the kinematic tree defined in the Skinned Multi-Person Linear (SMPL) model, where the upper body and lower body share only one common ancestor node, bringing the potential of decoupled reconstruction. We propose a stratified approach to decouple the conventional full-body avatar reconstruction pipeline into two stages, with the reconstruction of the upper body first and a subsequent reconstruction of the lower body conditioned on the previous stage. To implement this straightforward idea, we leverage the latent diffusion model as a powerful probabilistic generator, and train it to follow the latent distribution of decoupled motions explored by a VQ-VAE encoder-decoder model. Extensive experiments on AMASS mocap dataset demonstrate our state-of-the-art performance in the reconstruction of full-body motions.