Adversarial Self-Supervised Learning for Semi-Supervised 3D Action Recognition

TL;DR

This paper introduces ASSL, a novel adversarial self-supervised learning framework that enhances semi-supervised 3D action recognition by aligning unlabeled and labeled data representations through neighbor relation exploration and adversarial regularization.

Contribution

It proposes a new ASSL framework that combines SSL and adversarial learning to improve 3D action recognition with limited labeled data.

Findings

Outperforms state-of-the-art semi-supervised methods on NTU and N-UCLA datasets.

Effectively aligns feature distributions of labeled and unlabeled data.

Enhances representation discrimination in semi-supervised 3D action recognition.

Abstract

We consider the problem of semi-supervised 3D action recognition which has been rarely explored before. Its major challenge lies in how to effectively learn motion representations from unlabeled data. Self-supervised learning (SSL) has been proved very effective at learning representations from unlabeled data in the image domain. However, few effective self-supervised approaches exist for 3D action recognition, and directly applying SSL for semi-supervised learning suffers from misalignment of representations learned from SSL and supervised learning tasks. To address these issues, we present Adversarial Self-Supervised Learning (ASSL), a novel framework that tightly couples SSL and the semi-supervised scheme via neighbor relation exploration and adversarial learning. Specifically, we design an effective SSL scheme to improve the discrimination capability of learned representations for 3D action recognition, through exploring the data relations within a neighborhood. We further propose an adversarial regularization to align the feature distributions of labeled and unlabeled samples. To demonstrate effectiveness of the proposed ASSL in semi-supervised 3D action recognition, we conduct extensive experiments on NTU and N-UCLA datasets. The results confirm its advantageous performance over state-of-the-art semi-supervised methods in the few label regime for 3D action recognition.