A Deeper Look into DeepCap

TL;DR

This paper introduces a novel deep learning method for monocular human performance capture that does not require 3D ground truth data, outperforming existing methods in quality and robustness.

Contribution

It presents a weakly supervised deep learning approach with a dual-network architecture for dense human performance capture from monocular images.

Findings

Outperforms state-of-the-art in quality and robustness

Operates without 3D ground truth annotations

Effective in dense space-time coherent geometry reconstruction

Abstract

Human performance capture is a highly important computer vision problem with many applications in movie production and virtual/augmented reality. Many previous performance capture approaches either required expensive multi-view setups or did not recover dense space-time coherent geometry with frame-to-frame correspondences. We propose a novel deep learning approach for monocular dense human performance capture. Our method is trained in a weakly supervised manner based on multi-view supervision completely removing the need for training data with 3D ground truth annotations. The network architecture is based on two separate networks that disentangle the task into a pose estimation and a non-rigid surface deformation step. Extensive qualitative and quantitative evaluations show that our approach outperforms the state of the art in terms of quality and robustness. This work is an extended version of DeepCap where we provide more detailed explanations, comparisons and results as well as applications.