Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Recognition

TL;DR

This paper introduces AimCLR, a novel contrastive learning framework for skeleton-based action recognition that leverages extreme augmentations and abundant information mining to learn more universal and robust action representations.

Contribution

AimCLR employs extreme augmentations, an energy-based attention module, a dual divergence loss, and nearest neighbors mining to enhance self-supervised skeleton action learning.

Findings

Achieves superior performance on NTU RGB+D 60, PKU-MMD, NTU RGB+D 120 datasets.

Outperforms state-of-the-art methods across various evaluation protocols.

Demonstrates the effectiveness of extreme augmentations and information mining in self-supervised learning.

Abstract

In recent years, self-supervised representation learning for skeleton-based action recognition has been developed with the advance of contrastive learning methods. The existing contrastive learning methods use normal augmentations to construct similar positive samples, which limits the ability to explore novel movement patterns. In this paper, to make better use of the movement patterns introduced by extreme augmentations, a Contrastive Learning framework utilizing Abundant Information Mining for self-supervised action Representation (AimCLR) is proposed. First, the extreme augmentations and the Energy-based Attention-guided Drop Module (EADM) are proposed to obtain diverse positive samples, which bring novel movement patterns to improve the universality of the learned representations. Second, since directly using extreme augmentations may not be able to boost the performance due to the drastic changes in original identity, the Dual Distributional Divergence Minimization Loss (D$^3$M Loss) is proposed to minimize the distribution divergence in a more gentle way. Third, the Nearest Neighbors Mining (NNM) is proposed to further expand positive samples to make the abundant information mining process more reasonable. Exhaustive experiments on NTU RGB+D 60, PKU-MMD, NTU RGB+D 120 datasets have verified that our AimCLR can significantly perform favorably against state-of-the-art methods under a variety of evaluation protocols with observed higher quality action representations. Our code is available at https://github.com/Levigty/AimCLR.