JADE: Joint-aware Latent Diffusion for 3D Human Generative Modeling

TL;DR

JADE introduces a joint-aware latent diffusion framework for 3D human modeling, enabling fine-grained control, interpretability, and high-quality generation by decomposing human bodies into skeletons and surface features.

Contribution

The paper proposes a novel joint-aware latent representation and a cascaded diffusion pipeline for improved 3D human shape generation and control.

Findings

High reconstruction accuracy on public datasets

Enhanced editing controllability

Superior generation quality compared to existing methods

Abstract

Generative modeling of 3D human bodies have been studied extensively in computer vision. The core is to design a compact latent representation that is both expressive and semantically interpretable, yet existing approaches struggle to achieve both requirements. In this work, we introduce JADE, a generative framework that learns the variations of human shapes with fined-grained control. Our key insight is a joint-aware latent representation that decomposes human bodies into skeleton structures, modeled by joint positions, and local surface geometries, characterized by features attached to each joint. This disentangled latent space design enables geometric and semantic interpretation, facilitating users with flexible controllability. To generate coherent and plausible human shapes under our proposed decomposition, we also present a cascaded pipeline where two diffusions are employed to model the distribution of skeleton structures and local surface geometries respectively. Extensive experiments are conducted on public datasets, where we demonstrate the effectiveness of JADE framework in multiple tasks in terms of autoencoding reconstruction accuracy, editing controllability and generation quality compared with existing methods.