H4D: Human 4D Modeling by Learning Neural Compositional Representation

TL;DR

H4D introduces a neural compositional framework for 4D human modeling that leverages SMPL priors, enabling accurate dynamic 3D reconstructions and applications like motion retargeting and prediction.

Contribution

The paper proposes a novel H4D representation combining SMPL parameters with learned latent codes and GRU architectures for effective 4D human modeling.

Findings

Accurately recovers dynamic human geometry and motion.

Effective for motion retargeting and future prediction.

Demonstrates superior performance over existing methods.

Abstract

Despite the impressive results achieved by deep learning based 3D reconstruction, the techniques of directly learning to model 4D human captures with detailed geometry have been less studied. This work presents a novel framework that can effectively learn a compact and compositional representation for dynamic human by exploiting the human body prior from the widely used SMPL parametric model. Particularly, our representation, named H4D, represents a dynamic 3D human over a temporal span with the SMPL parameters of shape and initial pose, and latent codes encoding motion and auxiliary information. A simple yet effective linear motion model is proposed to provide a rough and regularized motion estimation, followed by per-frame compensation for pose and geometry details with the residual encoded in the auxiliary code. Technically, we introduce novel GRU-based architectures to facilitate learning and improve the representation capability. Extensive experiments demonstrate our method is not only efficacy in recovering dynamic human with accurate motion and detailed geometry, but also amenable to various 4D human related tasks, including motion retargeting, motion completion and future prediction. Please check out the project page for video and code: https://boyanjiang.github.io/H4D/.