Facial Expression Classification using Fusion of Deep Neural Network in Video for the 3rd ABAW3 Competition

TL;DR

This paper presents a deep neural network fusion approach with transformer encoding for facial expression classification in videos, achieving notable F1 scores in the ABAW3 competition.

Contribution

It introduces a transformer-based fusion method for robust facial expression representation in videos, advancing emotion recognition accuracy.

Findings

Achieved 30.35% F1 score on validation set.

Achieved 28.60% F1 score on test set.

Demonstrated effectiveness on the Aff-Wild2 dataset.

Abstract

For computers to recognize human emotions, expression classification is an equally important problem in the human-computer interaction area. In the 3rd Affective Behavior Analysis In-The-Wild competition, the task of expression classification includes eight classes with six basic expressions of human faces from videos. In this paper, we employ a transformer mechanism to encode the robust representation from the backbone. Fusion of the robust representations plays an important role in the expression classification task. Our approach achieves 30.35\% and 28.60\% for the $F_1$ score on the validation set and the test set, respectively. This result shows the effectiveness of the proposed architecture based on the Aff-Wild2 dataset.