4D Association Graph for Realtime Multi-person Motion Capture Using Multiple Video Cameras

TL;DR

This paper introduces a real-time multi-person motion capture system using a novel 4D association graph that unifies parsing, matching, and tracking across multiple views and time, achieving high accuracy and efficiency.

Contribution

It presents the first unified 4D association graph framework for real-time multi-view, multi-person motion capture, with a new efficient optimization method and a multiview dataset.

Findings

Runs at 30fps with 5 cameras on 5 persons

Outperforms state-of-the-art methods quantitatively

Robust to noisy detections

Abstract

This paper contributes a novel realtime multi-person motion capture algorithm using multiview video inputs. Due to the heavy occlusions in each view, joint optimization on the multiview images and multiple temporal frames is indispensable, which brings up the essential challenge of realtime efficiency. To this end, for the first time, we unify per-view parsing, cross-view matching, and temporal tracking into a single optimization framework, i.e., a 4D association graph that each dimension (image space, viewpoint and time) can be treated equally and simultaneously. To solve the 4D association graph efficiently, we further contribute the idea of 4D limb bundle parsing based on heuristic searching, followed with limb bundle assembling by proposing a bundle Kruskal's algorithm. Our method enables a realtime online motion capture system running at 30fps using 5 cameras on a 5-person scene. Benefiting from the unified parsing, matching and tracking constraints, our method is robust to noisy detection, and achieves high-quality online pose reconstruction quality. The proposed method outperforms the state-of-the-art method quantitatively without using high-level appearance information. We also contribute a multiview video dataset synchronized with a marker-based motion capture system for scientific evaluation.