The Best of Both Worlds: Combining Model-based and Nonparametric Approaches for 3D Human Body Estimation

TL;DR

This paper introduces a three-module framework that combines nonparametric and model-based methods for more accurate and robust 3D human body estimation from monocular images, especially under occlusion.

Contribution

The proposed framework integrates dense UV mapping, inverse kinematics, and UV inpainting to improve 3D human body reconstruction accuracy and robustness against occlusion.

Findings

Outperforms existing methods on multiple benchmarks.

Robust to partial occlusion.

Improves alignment between mesh and image evidence.

Abstract

Nonparametric based methods have recently shown promising results in reconstructing human bodies from monocular images while model-based methods can help correct these estimates and improve prediction. However, estimating model parameters from global image features may lead to noticeable misalignment between the estimated meshes and image evidence. To address this issue and leverage the best of both worlds, we propose a framework of three consecutive modules. A dense map prediction module explicitly establishes the dense UV correspondence between the image evidence and each part of the body model. The inverse kinematics module refines the key point prediction and generates a posed template mesh. Finally, a UV inpainting module relies on the corresponding feature, prediction and the posed template, and completes the predictions of occluded body shape. Our framework leverages the best of non-parametric and model-based methods and is also robust to partial occlusion. Experiments demonstrate that our framework outperforms existing 3D human estimation methods on multiple public benchmarks.