Imposing Temporal Consistency on Deep Monocular Body Shape and Pose Estimation

TL;DR

This paper introduces a method for improving the accuracy and temporal consistency of 3D human body shape and pose estimation from monocular image sequences, enhancing applications like sign language analysis.

Contribution

It presents a novel approach that integrates temporal constraints into the fitting process, ensuring consistent and robust modeling of body motion over time.

Findings

Improved accuracy in 3D body shape and pose estimation.

Enhanced temporal consistency in motion modeling.

Effective application to sign language analysis.

Abstract

Accurate and temporally consistent modeling of human bodies is essential for a wide range of applications, including character animation, understanding human social behavior and AR/VR interfaces. Capturing human motion accurately from a monocular image sequence is still challenging and the modeling quality is strongly influenced by the temporal consistency of the captured body motion. Our work presents an elegant solution for the integration of temporal constraints in the fitting process. This does not only increase temporal consistency but also robustness during the optimization. In detail, we derive parameters of a sequence of body models, representing shape and motion of a person, including jaw poses, facial expressions, and finger poses. We optimize these parameters over the complete image sequence, fitting one consistent body shape while imposing temporal consistency on the body motion, assuming linear body joint trajectories over a short time. Our approach enables the derivation of realistic 3D body models from image sequences, including facial expression and articulated hands. In extensive experiments, we show that our approach results in accurately estimated body shape and motion, also for challenging movements and poses. Further, we apply it to the special application of sign language analysis, where accurate and temporal consistent motion modelling is essential, and show that the approach is well-suited for this kind of application.