Self-supervised Gait-based Emotion Representation Learning from Selective Strongly Augmented Skeleton Sequences

TL;DR

This paper introduces a self-supervised contrastive learning framework with selective strong augmentation for gait-based emotion recognition, improving representation robustness from limited data.

Contribution

It proposes a novel SSA method and a feature fusion network to enhance gait emotion representations in a self-supervised manner.

Findings

Outperforms state-of-the-art on E-Gait and Emilya datasets.

Generates more diverse positive samples for better feature learning.

Achieves robust emotion recognition with limited labeled data.

Abstract

Emotion recognition is an important part of affective computing. Extracting emotional cues from human gaits yields benefits such as natural interaction, a nonintrusive nature, and remote detection. Recently, the introduction of self-supervised learning techniques offers a practical solution to the issues arising from the scarcity of labeled data in the field of gait-based emotion recognition. However, due to the limited diversity of gaits and the incompleteness of feature representations for skeletons, the existing contrastive learning methods are usually inefficient for the acquisition of gait emotions. In this paper, we propose a contrastive learning framework utilizing selective strong augmentation (SSA) for self-supervised gait-based emotion representation, which aims to derive effective representations from limited labeled gait data. First, we propose an SSA method for the gait emotion recognition task, which includes upper body jitter and random spatiotemporal mask. The goal of SSA is to generate more diverse and targeted positive samples and prompt the model to learn more distinctive and robust feature representations. Then, we design a complementary feature fusion network (CFFN) that facilitates the integration of cross-domain information to acquire topological structural and global adaptive features. Finally, we implement the distributional divergence minimization loss to supervise the representation learning of the generally and strongly augmented queries. Our approach is validated on the Emotion-Gait (E-Gait) and Emilya datasets and outperforms the state-of-the-art methods under different evaluation protocols.