Representation-Centric Survey of Skeletal Action Recognition and the ANUBIS Benchmark

TL;DR

This paper provides a comprehensive survey of skeleton-based action recognition methods, introduces the ANUBIS benchmark dataset to address real-world challenges, and analyzes how different feature representations impact recognition performance.

Contribution

It offers a systematic categorization of input features, introduces the ANUBIS dataset with complex scenarios, and benchmarks state-of-the-art models to reveal feature-performance relationships.

Findings

Strong dependency of recognition accuracy on feature types

Na"ive multi-representational fusion has limitations

Need for task-aware, semantically aligned integration strategies

Abstract

3D skeleton-based human action recognition has emerged as a powerful alternative to traditional RGB and depth-based approaches, offering robustness to environmental variations, computational efficiency, and enhanced privacy. Despite remarkable progress, current research remains fragmented across diverse input representations and lacks evaluation under scenarios that reflect modern real-world challenges. This paper presents a representation-centric survey of skeleton-based action recognition, systematically categorizing state-of-the-art methods by their input feature types: joint coordinates, bone vectors, motion flows, and extended representations, and analyzing how these choices influence spatial-temporal modeling strategies. Building on the insights from this review, we introduce ANUBIS, a large-scale, challenging skeleton action dataset designed to address critical gaps in existing benchmarks. ANUBIS incorporates multi-view recordings with back-view perspectives, complex multi-person interactions, fine-grained and violent actions, and contemporary social behaviors. We benchmark a diverse set of state-of-the-art models on ANUBIS and conduct an in-depth analysis of how different feature types affect recognition performance across 102 action categories. Our results show strong action-feature dependencies, highlight the limitations of na\"ive multi-representational fusion, and point toward the need for task-aware, semantically aligned integration strategies. This work offers both a comprehensive foundation and a practical benchmarking resource, aiming to guide the next generation of robust, generalizable skeleton-based action recognition systems for complex real-world scenarios. The dataset website, benchmarking framework, and download link are available at https://yliu1082.github.io/ANUBIS/.