FreqMixFormerV2: Lightweight Frequency-aware Mixed Transformer for Human Skeleton Action Recognition

TL;DR

FreqMixFormerV2 introduces a lightweight, frequency-aware transformer architecture for human skeleton action recognition, achieving high accuracy with significantly fewer parameters suitable for resource-limited environments.

Contribution

It presents a novel frequency-aware design and simplified attention mechanism that reduce model complexity while maintaining or improving recognition performance.

Findings

Outperforms state-of-the-art methods on standard datasets.

Uses only 60% of the parameters of comparable models.

Achieves a superior efficiency-accuracy trade-off.

Abstract

Transformer-based human skeleton action recognition has been developed for years. However, the complexity and high parameter count demands of these models hinder their practical applications, especially in resource-constrained environments. In this work, we propose FreqMixForemrV2, which was built upon the Frequency-aware Mixed Transformer (FreqMixFormer) for identifying subtle and discriminative actions with pioneered frequency-domain analysis. We design a lightweight architecture that maintains robust performance while significantly reducing the model complexity. This is achieved through a redesigned frequency operator that optimizes high-frequency and low-frequency parameter adjustments, and a simplified frequency-aware attention module. These improvements result in a substantial reduction in model parameters, enabling efficient deployment with only a minimal sacrifice in accuracy. Comprehensive evaluations of standard datasets (NTU RGB+D, NTU RGB+D 120, and NW-UCLA datasets) demonstrate that the proposed model achieves a superior balance between efficiency and accuracy, outperforming state-of-the-art methods with only 60% of the parameters.