FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration

TL;DR

FrankMocap is a fast, unified system for monocular 3D hand and body motion capture that operates in near real-time with improved accuracy, effectively capturing complex human motions in challenging in-the-wild scenarios.

Contribution

The paper introduces FrankMocap, a novel system that simultaneously captures 3D hand and body motion from monocular videos with high speed and accuracy, integrating state-of-the-art hand and body models.

Findings

Achieves 9.5 fps in monocular 3D motion capture.

Outperforms previous methods on public benchmarks.

Produces high-quality whole-body motion in real-world scenes.

Abstract

Although the essential nuance of human motion is often conveyed as a combination of body movements and hand gestures, the existing monocular motion capture approaches mostly focus on either body motion capture only ignoring hand parts or hand motion capture only without considering body motion. In this paper, we present FrankMocap, a motion capture system that can estimate both 3D hand and body motion from in-the-wild monocular inputs with faster speed (9.5 fps) and better accuracy than previous work. Our method works in near real-time (9.5 fps) and produces 3D body and hand motion capture outputs as a unified parametric model structure. Our method aims to capture 3D body and hand motion simultaneously from challenging in-the-wild monocular videos. To construct FrankMocap, we build the state-of-the-art monocular 3D "hand" motion capture method by taking the hand part of the whole body parametric model (SMPL-X). Our 3D hand motion capture output can be efficiently integrated to monocular body motion capture output, producing whole body motion results in a unified parrametric model structure. We demonstrate the state-of-the-art performance of our hand motion capture system in public benchmarks, and show the high quality of our whole body motion capture result in various challenging real-world scenes, including a live demo scenario.