Evolving Skeletons: Motion Dynamics in Action Recognition

TL;DR

This paper evaluates traditional and motion-enhanced skeleton representations using graph and hypergraph models for action recognition, revealing their strengths and limitations on benchmark datasets.

Contribution

It provides a comprehensive comparison of skeletal and hypergraph models with and without motion enhancement, highlighting the potential and challenges of Taylor-transformed skeletons.

Findings

Taylor-transformed skeletons improve motion dynamics representation

Hypergraph models capture higher-order joint interactions

Challenges remain in fully leveraging motion-enhanced skeletons

Abstract

Skeleton-based action recognition has gained significant attention for its ability to efficiently represent spatiotemporal information in a lightweight format. Most existing approaches use graph-based models to process skeleton sequences, where each pose is represented as a skeletal graph structured around human physical connectivity. Among these, the Spatiotemporal Graph Convolutional Network (ST-GCN) has become a widely used framework. Alternatively, hypergraph-based models, such as the Hyperformer, capture higher-order correlations, offering a more expressive representation of complex joint interactions. A recent advancement, termed Taylor Videos, introduces motion-enhanced skeleton sequences by embedding motion concepts, providing a fresh perspective on interpreting human actions in skeleton-based action recognition. In this paper, we conduct a comprehensive evaluation of both traditional skeleton sequences and Taylor-transformed skeletons using ST-GCN and Hyperformer models on the NTU-60 and NTU-120 datasets. We compare skeletal graph and hypergraph representations, analyzing static poses against motion-injected poses. Our findings highlight the strengths and limitations of Taylor-transformed skeletons, demonstrating their potential to enhance motion dynamics while exposing current challenges in fully using their benefits. This study underscores the need for innovative skeletal modelling techniques to effectively handle motion-rich data and advance the field of action recognition.