Efficient Labelling of Affective Video Datasets via Few-Shot & Multi-Task Contrastive Learning

TL;DR

This paper introduces MT-CLAR, a multi-task contrastive learning framework that enables effective affective video labeling with minimal labeled data, achieving comparable or superior results to state-of-the-art methods.

Contribution

The paper presents a novel multi-task contrastive learning approach for few-shot affect inference and automated video labeling, reducing the need for extensive labeled datasets.

Findings

MT-CLAR achieves state-of-the-art performance on affect prediction.

Significant performance gains with only 6% of labeled data.

Effective extension from facial images to video affect labeling.

Abstract

Whilst deep learning techniques have achieved excellent emotion prediction, they still require large amounts of labelled training data, which are (a) onerous and tedious to compile, and (b) prone to errors and biases. We propose Multi-Task Contrastive Learning for Affect Representation (\textbf{MT-CLAR}) for few-shot affect inference. MT-CLAR combines multi-task learning with a Siamese network trained via contrastive learning to infer from a pair of expressive facial images (a) the (dis)similarity between the facial expressions, and (b) the difference in valence and arousal levels of the two faces. We further extend the image-based MT-CLAR framework for automated video labelling where, given one or a few labelled video frames (termed \textit{support-set}), MT-CLAR labels the remainder of the video for valence and arousal. Experiments are performed on the AFEW-VA dataset with multiple support-set configurations; moreover, supervised learning on representations learnt via MT-CLAR are used for valence, arousal and categorical emotion prediction on the AffectNet and AFEW-VA datasets. The results show that valence and arousal predictions via MT-CLAR are very comparable to the state-of-the-art (SOTA), and we significantly outperform SOTA with a support-set $\approx$6\% the size of the video dataset.