UniSTFormer: Unified Spatio-Temporal Lightweight Transformer for Efficient Skeleton-Based Action Recognition

TL;DR

UniSTFormer introduces a unified lightweight transformer for skeleton-based action recognition that combines spatial and temporal modeling in a single module, significantly reducing complexity while maintaining high accuracy.

Contribution

The paper presents a novel unified spatio-temporal transformer framework with a simplified multi-scale pooling module, improving efficiency and scalability over existing complex models.

Findings

Reduces parameter count by over 58%.

Lowers computational cost by over 60%.

Maintains competitive recognition accuracy.

Abstract

Skeleton-based action recognition (SAR) has achieved impressive progress with transformer architectures. However, existing methods often rely on complex module compositions and heavy designs, leading to increased parameter counts, high computational costs, and limited scalability. In this paper, we propose a unified spatio-temporal lightweight transformer framework that integrates spatial and temporal modeling within a single attention module, eliminating the need for separate temporal modeling blocks. This approach reduces redundant computations while preserving temporal awareness within the spatial modeling process. Furthermore, we introduce a simplified multi-scale pooling fusion module that combines local and global pooling pathways to enhance the model's ability to capture fine-grained local movements and overarching global motion patterns. Extensive experiments on benchmark datasets demonstrate that our lightweight model achieves a superior balance between accuracy and efficiency, reducing parameter complexity by over 58% and lowering computational cost by over 60% compared to state-of-the-art transformer-based baselines, while maintaining competitive recognition performance.