Every Angle Is Worth A Second Glance: Mining Kinematic Skeletal Structures from Multi-view Joint Cloud

TL;DR

This paper introduces a novel transformer-based framework, JCSAT, for multi-view multi-person 3D motion capture that effectively handles occlusions and joint association by leveraging a comprehensive joint cloud and a new selection mechanism.

Contribution

The paper proposes the Joint Cloud Selection and Aggregation Transformer (JCSAT) with OTAP for improved 3D pose estimation from multi-view data, and introduces the BUMocap-X dataset for complex scenarios.

Findings

JCSAT outperforms existing methods on benchmark datasets.

The framework effectively handles severe occlusions and complex interactions.

The new dataset BUMocap-X provides a challenging benchmark for future research.

Abstract

Multi-person motion capture over sparse angular observations is a challenging problem under interference from both self- and mutual-occlusions. Existing works produce accurate 2D joint detection, however, when these are triangulated and lifted into 3D, available solutions all struggle in selecting the most accurate candidates and associating them to the correct joint type and target identity. As such, in order to fully utilize all accurate 2D joint location information, we propose to independently triangulate between all same-typed 2D joints from all camera views regardless of their target ID, forming the Joint Cloud. Joint Cloud consist of both valid joints lifted from the same joint type and target ID, as well as falsely constructed ones that are from different 2D sources. These redundant and inaccurate candidates are processed over the proposed Joint Cloud Selection and Aggregation Transformer (JCSAT) involving three cascaded encoders which deeply explore the trajectile, skeletal structural, and view-dependent correlations among all 3D point candidates in the cross-embedding space. An Optimal Token Attention Path (OTAP) module is proposed which subsequently selects and aggregates informative features from these redundant observations for the final prediction of human motion. To demonstrate the effectiveness of JCSAT, we build and publish a new multi-person motion capture dataset BUMocap-X with complex interactions and severe occlusions. Comprehensive experiments over the newly presented as well as benchmark datasets validate the effectiveness of the proposed framework, which outperforms all existing state-of-the-art methods, especially under challenging occlusion scenarios.