NTU-X: An Enhanced Large-scale Dataset for Improving Pose-based Recognition of Subtle Human Actions

TL;DR

This paper introduces NTU-X datasets with detailed facial and finger joints to enhance skeleton-based human action recognition, addressing the lack of fine-grained joint data in existing datasets.

Contribution

The creation of NTU60-X and NTU120-X datasets with detailed facial and finger joints, and adapting models to utilize this richer skeleton data for improved recognition.

Findings

Improved recognition accuracy overall.

Enhanced performance on difficult action categories.

Effective use of detailed joint data in models.

Abstract

The lack of fine-grained joints (facial joints, hand fingers) is a fundamental performance bottleneck for state of the art skeleton action recognition models. Despite this bottleneck, community's efforts seem to be invested only in coming up with novel architectures. To specifically address this bottleneck, we introduce two new pose based human action datasets - NTU60-X and NTU120-X. Our datasets extend the largest existing action recognition dataset, NTU-RGBD. In addition to the 25 body joints for each skeleton as in NTU-RGBD, NTU60-X and NTU120-X dataset includes finger and facial joints, enabling a richer skeleton representation. We appropriately modify the state of the art approaches to enable training using the introduced datasets. Our results demonstrate the effectiveness of these NTU-X datasets in overcoming the aforementioned bottleneck and improve state of the art performance, overall and on previously worst performing action categories. Code and pretrained models can be found at https://github.com/skelemoa/ntu-x .