SS-MFAR : Semi-supervised Multi-task Facial Affect Recognition

TL;DR

This paper introduces SS-MFAR, a semi-supervised multi-task deep learning approach for facial affect recognition in real-world scenarios with incomplete labels, achieving improved performance on the ABAW 2022 challenge.

Contribution

It proposes a novel semi-supervised multi-task learning framework with adaptive thresholds for affect recognition on in-the-wild datasets.

Findings

Effective handling of incomplete labels in affect recognition

Improved multi-task learning performance

Open-source implementation available

Abstract

Automatic affect recognition has applications in many areas such as education, gaming, software development, automotives, medical care, etc. but it is non trivial task to achieve appreciable performance on in-the-wild data sets. In-the-wild data sets though represent real-world scenarios better than synthetic data sets, the former ones suffer from the problem of incomplete labels. Inspired by semi-supervised learning, in this paper, we introduce our submission to the Multi-Task-Learning Challenge at the 4th Affective Behavior Analysis in-the-wild (ABAW) 2022 Competition. The three tasks that are considered in this challenge are valence-arousal(VA) estimation, classification of expressions into 6 basic (anger, disgust, fear, happiness, sadness, surprise), neutral, and the 'other' category and 12 action units(AU) numbered AU-{1,2,4,6,7,10,12,15,23,24,25,26}. Our method Semi-supervised Multi-task Facial Affect Recognition titled SS-MFAR uses a deep residual network with task specific classifiers for each of the tasks along with adaptive thresholds for each expression class and semi-supervised learning for the incomplete labels. Source code is available at https://github.com/1980x/ABAW2022DMACS.