Multi Modal Facial Expression Recognition with Transformer-Based Fusion Networks and Dynamic Sampling

TL;DR

This paper presents a multi-modal facial expression recognition approach that combines audio and visual data using transformer-based fusion and dynamic sampling to improve accuracy in challenging datasets.

Contribution

It introduces a novel Modal Fusion Module with Swin Transformer features and addresses data imbalance through dynamic resampling, advancing multi-modal emotion recognition.

Findings

Effective fusion of audio and visual features improves recognition accuracy.

Dynamic sampling mitigates dataset imbalance issues.

Model performs well in ABAW challenge evaluations.

Abstract

Facial expression recognition is an essential task for various applications, including emotion detection, mental health analysis, and human-machine interactions. In this paper, we propose a multi-modal facial expression recognition method that exploits audio information along with facial images to provide a crucial clue to differentiate some ambiguous facial expressions. Specifically, we introduce a Modal Fusion Module (MFM) to fuse audio-visual information, where image and audio features are extracted from Swin Transformer. Additionally, we tackle the imbalance problem in the dataset by employing dynamic data resampling. Our model has been evaluated in the Affective Behavior in-the-wild (ABAW) challenge of CVPR 2023.