HULC: 3D Human Motion Capture with Pose Manifold Sampling and Dense Contact Guidance

TL;DR

HULC is a novel monocular 3D human motion capture method that incorporates scene geometry and dense contact information to produce more accurate and physically plausible human pose estimations, outperforming existing approaches.

Contribution

HULC introduces scene-aware pose estimation with dense contact guidance and pose manifold sampling, improving accuracy and realism in monocular 3D human MoCap.

Findings

Outperforms existing methods in accuracy and plausibility

Reduces body-scene inter-penetrations and jitter

Produces more physically consistent 3D poses

Abstract

Marker-less monocular 3D human motion capture (MoCap) with scene interactions is a challenging research topic relevant for extended reality, robotics and virtual avatar generation. Due to the inherent depth ambiguity of monocular settings, 3D motions captured with existing methods often contain severe artefacts such as incorrect body-scene inter-penetrations, jitter and body floating. To tackle these issues, we propose HULC, a new approach for 3D human MoCap which is aware of the scene geometry. HULC estimates 3D poses and dense body-environment surface contacts for improved 3D localisations, as well as the absolute scale of the subject. Furthermore, we introduce a 3D pose trajectory optimisation based on a novel pose manifold sampling that resolves erroneous body-environment inter-penetrations. Although the proposed method requires less structured inputs compared to existing scene-aware monocular MoCap algorithms, it produces more physically-plausible poses: HULC significantly and consistently outperforms the existing approaches in various experiments and on different metrics. Project page: https://vcai.mpi-inf.mpg.de/projects/HULC/.