Human Performance Capture from Monocular Video in the Wild

TL;DR

This paper introduces a monocular video-based method for capturing dynamic 3D human shapes in challenging poses without specialized equipment, advancing robustness and applicability in real-world scenarios.

Contribution

It presents a novel approach that builds a 3D human template and tracks its deformation from monocular videos, outperforming existing methods in wild conditions.

Findings

Outperforms state-of-the-art on 3DPW dataset

Demonstrates robustness on iPER videos

Effective in challenging body poses

Abstract

Capturing the dynamically deforming 3D shape of clothed human is essential for numerous applications, including VR/AR, autonomous driving, and human-computer interaction. Existing methods either require a highly specialized capturing setup, such as expensive multi-view imaging systems, or they lack robustness to challenging body poses. In this work, we propose a method capable of capturing the dynamic 3D human shape from a monocular video featuring challenging body poses, without any additional input. We first build a 3D template human model of the subject based on a learned regression model. We then track this template model's deformation under challenging body articulations based on 2D image observations. Our method outperforms state-of-the-art methods on an in-the-wild human video dataset 3DPW. Moreover, we demonstrate its efficacy in robustness and generalizability on videos from iPER datasets.