Leveraging Recent Advances in Deep Learning for Audio-Visual Emotion Recognition

TL;DR

This paper introduces a deep learning framework for multi-modal emotion recognition that combines recent techniques like knowledge distillation and model-level fusion, achieving superior results on multiple datasets.

Contribution

It presents a novel deep learning approach that effectively fuses audio-visual features and captures temporal dynamics for improved emotion recognition accuracy.

Findings

Outperforms state-of-the-art in valence prediction on RECOLA dataset.

Achieves superior facial expression recognition on AffectNet and Google Facial Expression Comparison datasets.

Demonstrates the effectiveness of model-level fusion and deep architectures in emotion recognition.

Abstract

Emotional expressions are the behaviors that communicate our emotional state or attitude to others. They are expressed through verbal and non-verbal communication. Complex human behavior can be understood by studying physical features from multiple modalities; mainly facial, vocal and physical gestures. Recently, spontaneous multi-modal emotion recognition has been extensively studied for human behavior analysis. In this paper, we propose a new deep learning-based approach for audio-visual emotion recognition. Our approach leverages recent advances in deep learning like knowledge distillation and high-performing deep architectures. The deep feature representations of the audio and visual modalities are fused based on a model-level fusion strategy. A recurrent neural network is then used to capture the temporal dynamics. Our proposed approach substantially outperforms state-of-the-art approaches in predicting valence on the RECOLA dataset. Moreover, our proposed visual facial expression feature extraction network outperforms state-of-the-art results on the AffectNet and Google Facial Expression Comparison datasets.