A Dataset and Evaluation for Complex 4D Markerless Human Motion Capture

TL;DR

This paper introduces a comprehensive dataset and evaluation framework for complex 4D markerless human motion capture, emphasizing multi-person interactions, occlusions, and real-world challenges.

Contribution

The authors present a new dataset with multi-view RGB and depth data, ground-truth 3D motion, and challenging scenarios to benchmark and improve markerless human motion capture models.

Findings

State-of-the-art models perform poorly under realistic conditions.

Fine-tuning on the new dataset improves model generalization.

The dataset reveals critical gaps in current markerless MoCap approaches.

Abstract

Marker-based motion capture (MoCap) systems have long been the gold standard for accurate 4D human modeling, yet their reliance on specialized hardware and markers limits scalability and real-world deployment. Advancing reliable markerless 4D human motion capture requires datasets that reflect the complexity of real-world human interactions. Yet, existing benchmarks often lack realistic multi-person dynamics, severe occlusions, and challenging interaction patterns, leading to a persistent domain gap. In this work, we present a new dataset and evaluation for complex 4D markerless human motion capture. Our proposed MoCap dataset captures both single and multi-person scenarios with intricate motions, frequent inter-person occlusions, rapid position exchanges between similarly dressed subjects, and varying subject distances. It includes synchronized multi-view RGB and depth sequences, accurate camera calibration, ground-truth 3D motion capture from a Vicon system, and corresponding SMPL/SMPL-X parameters. This setup ensures precise alignment between visual observations and motion ground truth. Benchmarking state-of-the-art markerless MoCap models reveals substantial performance degradation under these realistic conditions, highlighting limitations of current approaches. We further demonstrate that targeted fine-tuning improves generalization, validating the dataset's realism and value for model development. Our evaluation exposes critical gaps in existing models and provides a rigorous foundation for advancing robust markerless 4D human motion capture.