PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding

TL;DR

PKU-MMD is a comprehensive large-scale benchmark dataset for continuous multi-modal 3D human action understanding, featuring diverse activities, multiple modalities, and extensive annotations to advance research in action detection.

Contribution

This paper introduces PKU-MMD, a new large-scale, multi-modality dataset for continuous 3D human action understanding, filling a gap in existing benchmarks focused on segmented videos.

Findings

Extensive experiments demonstrate the dataset's utility across various modalities and scenarios.

Evaluation protocols and metrics, including a new 2D-AP, are proposed for benchmarking.

The dataset covers 51 actions with nearly 20,000 instances, supporting advanced research.

Abstract

Despite the fact that many 3D human activity benchmarks being proposed, most existing action datasets focus on the action recognition tasks for the segmented videos. There is a lack of standard large-scale benchmarks, especially for current popular data-hungry deep learning based methods. In this paper, we introduce a new large scale benchmark (PKU-MMD) for continuous multi-modality 3D human action understanding and cover a wide range of complex human activities with well annotated information. PKU-MMD contains 1076 long video sequences in 51 action categories, performed by 66 subjects in three camera views. It contains almost 20,000 action instances and 5.4 million frames in total. Our dataset also provides multi-modality data sources, including RGB, depth, Infrared Radiation and Skeleton. With different modalities, we conduct extensive experiments on our dataset in terms of two scenarios and evaluate different methods by various metrics, including a new proposed evaluation protocol 2D-AP. We believe this large-scale dataset will benefit future researches on action detection for the community.