Interpretable Deep Neural Networks for Dimensional and Categorical Emotion Recognition in-the-wild

TL;DR

This paper develops interpretable deep neural networks for recognizing emotions in-the-wild using combined categorical and valence-arousal representations, improving performance and interpretability.

Contribution

It introduces a combined training approach for CNN+RNN models using both emotion categories and valence-arousal, enhancing recognition accuracy and interpretability.

Findings

Combined models outperform single-representation models.

Mapping between emotion categories and valence-arousal explains performance gains.

Interpretable neural networks reveal relationships between emotion representations.

Abstract

Emotions play an important role in people's life. Understanding and recognising is not only important for interpersonal communication, but also has promising applications in Human-Computer Interaction, automobile safety and medical research. This project focuses on extending the emotion recognition database, and training the CNN + RNN emotion recognition neural networks with emotion category representation and valence \& arousal representation. The combined models are constructed by training the two representations simultaneously. The comparison and analysis between the three types of model are discussed. The inner-relationship between two emotion representations and the interpretability of the neural networks are investigated. The findings suggest that categorical emotion recognition performance can benefit from training with a combined model. And the mapping of emotion category and valence \& arousal values can explain this phenomenon.