Look Ma, no markers: holistic performance capture without the hassle

TL;DR

This paper introduces a novel marker-free method for holistic human performance capture that works across face, body, and hands using arbitrary cameras and environments, without manual setup.

Contribution

It presents the first comprehensive technique for markerless, high-quality 3D reconstruction of the entire human body, including detailed facial features, without calibration or custom hardware.

Findings

Achieves state-of-the-art results on multiple benchmarks.

Generalizes well across diverse datasets and environments.

Operates with arbitrary camera rigs and clothing.

Abstract

We tackle the problem of highly-accurate, holistic performance capture for the face, body and hands simultaneously. Motion-capture technologies used in film and game production typically focus only on face, body or hand capture independently, involve complex and expensive hardware and a high degree of manual intervention from skilled operators. While machine-learning-based approaches exist to overcome these problems, they usually only support a single camera, often operate on a single part of the body, do not produce precise world-space results, and rarely generalize outside specific contexts. In this work, we introduce the first technique for marker-free, high-quality reconstruction of the complete human body, including eyes and tongue, without requiring any calibration, manual intervention or custom hardware. Our approach produces stable world-space results from arbitrary camera rigs as well as supporting varied capture environments and clothing. We achieve this through a hybrid approach that leverages machine learning models trained exclusively on synthetic data and powerful parametric models of human shape and motion. We evaluate our method on a number of body, face and hand reconstruction benchmarks and demonstrate state-of-the-art results that generalize on diverse datasets.