Facial Affect Recognition based on Multi Architecture Encoder and Feature Fusion for the ABAW7 Challenge

TL;DR

This paper introduces a multi-architecture encoder with feature fusion and an affine module to improve facial affect recognition across three ABAW7 sub-challenges, achieving significant performance gains.

Contribution

The novel integration of a Transformer Encoder with feature fusion and an affine module for better feature alignment in facial affect recognition.

Findings

Significant performance improvements over baselines.

Effective feature alignment across different modalities.

Successful application to ABAW7 challenges.

Abstract

In this paper, we present our approach to addressing the challenges of the 7th ABAW competition. The competition comprises three sub-challenges: Valence Arousal (VA) estimation, Expression (Expr) classification, and Action Unit (AU) detection. To tackle these challenges, we employ state-of-the-art models to extract powerful visual features. Subsequently, a Transformer Encoder is utilized to integrate these features for the VA, Expr, and AU sub-challenges. To mitigate the impact of varying feature dimensions, we introduce an affine module to align the features to a common dimension. Overall, our results significantly outperform the baselines.