SS-VAERR: Self-Supervised Apparent Emotional Reaction Recognition from Video

TL;DR

This paper introduces a self-supervised learning approach for recognizing apparent emotional reactions from videos, leveraging pretext tasks and loss combinations to achieve state-of-the-art results in emotion recognition.

Contribution

It provides a comprehensive analysis of pretext tasks and loss functions for self-supervised video-based emotion recognition, advancing the state-of-the-art performance.

Findings

Self-supervised pretext tasks improve emotion recognition accuracy.

Combining regression and classification losses enhances performance.

Achieved current state-of-the-art results on spontaneous emotion datasets.

Abstract

This work focuses on the apparent emotional reaction recognition (AERR) from the video-only input, conducted in a self-supervised fashion. The network is first pre-trained on different self-supervised pretext tasks and later fine-tuned on the downstream target task. Self-supervised learning facilitates the use of pre-trained architectures and larger datasets that might be deemed unfit for the target task and yet might be useful to learn informative representations and hence provide useful initializations for further fine-tuning on smaller more suitable data. Our presented contribution is two-fold: (1) an analysis of different state-of-the-art (SOTA) pretext tasks for the video-only apparent emotional reaction recognition architecture, and (2) an analysis of various combinations of the regression and classification losses that are likely to improve the performance further. Together these two contributions result in the current state-of-the-art performance for the video-only spontaneous apparent emotional reaction recognition with continuous annotations.