Deep3DPose: Realtime Reconstruction of Arbitrarily Posed Human Bodies from Single RGB Images

TL;DR

Deep3DPose presents a real-time deep learning system that reconstructs detailed 3D human body shapes and poses from single RGB images, combining multi-task CNN predictions with optimization for high accuracy.

Contribution

It introduces a novel multi-task deep learning framework for simultaneous prediction of multiple cues, improving 3D human reconstruction from single images in real-time.

Findings

Achieves accurate real-time 3D human pose and shape reconstruction.

Outperforms state-of-the-art methods on challenging in-the-wild images.

Demonstrates robustness and high accuracy in diverse scenarios.

Abstract

We introduce an approach that accurately reconstructs 3D human poses and detailed 3D full-body geometric models from single images in realtime. The key idea of our approach is a novel end-to-end multi-task deep learning framework that uses single images to predict five outputs simultaneously: foreground segmentation mask, 2D joints positions, semantic body partitions, 3D part orientations and uv coordinates (uv map). The multi-task network architecture not only generates more visual cues for reconstruction, but also makes each individual prediction more accurate. The CNN regressor is further combined with an optimization based algorithm for accurate kinematic pose reconstruction and full-body shape modeling. We show that the realtime reconstruction reaches accurate fitting that has not been seen before, especially for wild images. We demonstrate the results of our realtime 3D pose and human body reconstruction system on various challenging in-the-wild videos. We show the system advances the frontier of 3D human body and pose reconstruction from single images by quantitative evaluations and comparisons with state-of-the-art methods.