Balanced Representation Learning for Long-tailed Skeleton-based Action Recognition

TL;DR

This paper introduces a balanced representation learning approach for long-tailed skeleton-based action recognition, effectively addressing data imbalance by expanding samples, mitigating bias, and leveraging complementary information, leading to improved performance and generalization.

Contribution

The paper proposes a novel method combining spatial-temporal exploration, a detached learning schedule, and skip-modal features to improve long-tailed action recognition.

Findings

Significant performance improvements over SOTA on four datasets.

Effective mitigation of class imbalance in skeleton-based action recognition.

Enhanced generalization demonstrated through extensive experiments.

Abstract

Skeleton-based action recognition has recently made significant progress. However, data imbalance is still a great challenge in real-world scenarios. The performance of current action recognition algorithms declines sharply when training data suffers from heavy class imbalance. The imbalanced data actually degrades the representations learned by these methods and becomes the bottleneck for action recognition. How to learn unbiased representations from imbalanced action data is the key to long-tailed action recognition. In this paper, we propose a novel balanced representation learning method to address the long-tailed problem in action recognition. Firstly, a spatial-temporal action exploration strategy is presented to expand the sample space effectively, generating more valuable samples in a rebalanced manner. Secondly, we design a detached action-aware learning schedule to further mitigate the bias in the representation space. The schedule detaches the representation learning of tail classes from training and proposes an action-aware loss to impose more effective constraints. Additionally, a skip-modal representation is proposed to provide complementary structural information. The proposed method is validated on four skeleton datasets, NTU RGB+D 60, NTU RGB+D 120, NW-UCLA, and Kinetics. It not only achieves consistently large improvement compared to the state-of-the-art (SOTA) methods, but also demonstrates a superior generalization capacity through extensive experiments. Our code is available at https://github.com/firework8/BRL.