Facial Affect Analysis: Learning from Synthetic Data & Multi-Task Learning Challenges

TL;DR

This paper introduces novel frameworks for facial affect analysis in unconstrained settings, addressing multi-task learning and synthetic data challenges, with experimental validation showing significant improvements over baselines.

Contribution

It proposes new methods for multi-task learning and synthetic data utilization in facial affect analysis, enhancing performance in real-world scenarios.

Findings

Outperformed baseline methods significantly

Effective handling of synthetic data issues

Improved multi-task learning performance

Abstract

Facial affect analysis remains a challenging task with its setting transitioned from lab-controlled to in-the-wild situations. In this paper, we present novel frameworks to handle the two challenges in the 4th Affective Behavior Analysis In-The-Wild (ABAW) competition: i) Multi-Task-Learning (MTL) Challenge and ii) Learning from Synthetic Data (LSD) Challenge. For MTL challenge, we adopt the SMM-EmotionNet with a better ensemble strategy of feature vectors. For LSD challenge, we propose respective methods to combat the problems of single labels, imbalanced distribution, fine-tuning limitations, and choice of model architectures. Experimental results on the official validation sets from the competition demonstrated that our proposed approaches outperformed baselines by a large margin. The code is available at https://github.com/sylyoung/ABAW4-HUST-ANT.