Affective Behavior Analysis using Action Unit Relation Graph and Multi-task Cross Attention

TL;DR

This paper proposes a novel multi-task learning approach for affective behavior analysis that leverages an action unit relation graph and cross-attention to improve performance across emotion, age, and gender recognition tasks.

Contribution

It introduces a cross-attentive module and a facial graph to enhance multi-task learning for facial behavior analysis in-the-wild.

Findings

Achieved an evaluation score of 128.8, outperforming the baseline of 30.

Effectively integrated action unit relations to improve multi-task learning.

Demonstrated the effectiveness of cross-attention in multi-task facial analysis.

Abstract

Facial behavior analysis is a broad topic with various categories such as facial emotion recognition, age, and gender recognition. Many studies focus on individual tasks while the multi-task learning approach is still an open research issue and requires more research. In this paper, we present our solution and experiment result for the Multi-Task Learning challenge of the Affective Behavior Analysis in-the-wild competition. The challenge is a combination of three tasks: action unit detection, facial expression recognition, and valance-arousal estimation. To address this challenge, we introduce a cross-attentive module to improve multi-task learning performance. Additionally, a facial graph is applied to capture the association among action units. As a result, we achieve the evaluation measure of 128.8 on the validation data provided by the organizers, which outperforms the baseline result of 30.