PC-HMR: Pose Calibration for 3D Human Mesh Recovery from 2D Images/Videos

TL;DR

This paper introduces Pose Calibration frameworks that improve 3D human mesh recovery from 2D images/videos by effectively correcting mesh pose using 3D pose guidance, achieving state-of-the-art results without extra annotations.

Contribution

The paper proposes two novel Pose Calibration frameworks, Serial and Parallel PC-HMR, integrating 3D pose estimators with HMR to enhance mesh accuracy and handle bone length variations.

Findings

Achieved state-of-the-art results on Human3.6M, 3DPW, and SURREAL datasets.

Effectively corrects mesh pose without requiring additional 3D annotations during testing.

Flexible calibration handles bone length variations and complex activities.

Abstract

The end-to-end Human Mesh Recovery (HMR) approach has been successfully used for 3D body reconstruction. However, most HMR-based frameworks reconstruct human body by directly learning mesh parameters from images or videos, while lacking explicit guidance of 3D human pose in visual data. As a result, the generated mesh often exhibits incorrect pose for complex activities. To tackle this problem, we propose to exploit 3D pose to calibrate human mesh. Specifically, we develop two novel Pose Calibration frameworks, i.e., Serial PC-HMR and Parallel PC-HMR. By coupling advanced 3D pose estimators and HMR in a serial or parallel manner, these two frameworks can effectively correct human mesh with guidance of a concise pose calibration module. Furthermore, since the calibration module is designed via non-rigid pose transformation, our PC-HMR frameworks can flexibly tackle bone length variations to alleviate misplacement in the calibrated mesh. Finally, our frameworks are based on generic and complementary integration of data-driven learning and geometrical modeling. Via plug-and-play modules, they can be efficiently adapted for both image/video-based human mesh recovery. Additionally, they have no requirement of extra 3D pose annotations in the testing phase, which releases inference difficulties in practice. We perform extensive experiments on the popular bench-marks, i.e., Human3.6M, 3DPW and SURREAL, where our PC-HMR frameworks achieve the SOTA results.