Quo Vadis, Skeleton Action Recognition ?

TL;DR

This paper explores the challenges of skeleton-based human action recognition in real-world scenarios by introducing new datasets, benchmarking existing models, and analyzing their limitations across diverse and unconstrained actions.

Contribution

It introduces three novel datasets—Skeletics-152, Skeleton-Mimetics, and Metaphorics—and provides comprehensive benchmarking and analysis of current models on these datasets and NTU-120.

Findings

Existing models face challenges with in-the-wild actions

Significant domain gaps exist between datasets

New datasets enable exploration of complex action recognition

Abstract

In this paper, we study current and upcoming frontiers across the landscape of skeleton-based human action recognition. To study skeleton-action recognition in the wild, we introduce Skeletics-152, a curated and 3-D pose-annotated subset of RGB videos sourced from Kinetics-700, a large-scale action dataset. We extend our study to include out-of-context actions by introducing Skeleton-Mimetics, a dataset derived from the recently introduced Mimetics dataset. We also introduce Metaphorics, a dataset with caption-style annotated YouTube videos of the popular social game Dumb Charades and interpretative dance performances. We benchmark state-of-the-art models on the NTU-120 dataset and provide multi-layered assessment of the results. The results from benchmarking the top performers of NTU-120 on the newly introduced datasets reveal the challenges and domain gap induced by actions in the wild. Overall, our work characterizes the strengths and limitations of existing approaches and datasets. Via the introduced datasets, our work enables new frontiers for human action recognition.