HybridCap: Inertia-aid Monocular Capture of Challenging Human Motions

TL;DR

HybridCap is a novel monocular motion capture system that combines minimal inertial sensors with visual data, enabling accurate, real-time tracking of complex human movements that are challenging for traditional methods.

Contribution

The paper introduces HybridCap, a lightweight hybrid mocap approach using only 4 IMUs and a learning-optimization framework for robust, real-time 3D human motion capture.

Findings

Handles complex movements like dance and fitness actions.

Achieves real-time performance up to 60 fps.

Outperforms existing monocular mocap methods in accuracy.

Abstract

Monocular 3D motion capture (mocap) is beneficial to many applications. The use of a single camera, however, often fails to handle occlusions of different body parts and hence it is limited to capture relatively simple movements. We present a light-weight, hybrid mocap technique called HybridCap that augments the camera with only 4 Inertial Measurement Units (IMUs) in a learning-and-optimization framework. We first employ a weakly-supervised and hierarchical motion inference module based on cooperative Gated Recurrent Unit (GRU) blocks that serve as limb, body and root trackers as well as an inverse kinematics solver. Our network effectively narrows the search space of plausible motions via coarse-to-fine pose estimation and manages to tackle challenging movements with high efficiency. We further develop a hybrid optimization scheme that combines inertial feedback and visual cues to improve tracking accuracy. Extensive experiments on various datasets demonstrate HybridCap can robustly handle challenging movements ranging from fitness actions to Latin dance. It also achieves real-time performance up to 60 fps with state-of-the-art accuracy.