How Do You Perceive My Face? Recognizing Facial Expressions in Multi-Modal Context by Modeling Mental Representations

TL;DR

This paper introduces a novel multi-modal facial expression recognition model that incorporates mental representations and context, achieving state-of-the-art accuracy and providing visual insights into human perception processes.

Contribution

The model uniquely synthesizes mental representations and adapts to context using a VAE-GAN architecture with a new attention mechanism, advancing facial expression recognition.

Findings

Achieves 81.01% accuracy on RAVDESS dataset.

Achieves 79.34% accuracy on MEAD dataset.

Synthesized expressions closely match human mental representations.

Abstract

Facial expression perception in humans inherently relies on prior knowledge and contextual cues, contributing to efficient and flexible processing. For instance, multi-modal emotional context (such as voice color, affective text, body pose, etc.) can prompt people to perceive emotional expressions in objectively neutral faces. Drawing inspiration from this, we introduce a novel approach for facial expression classification that goes beyond simple classification tasks. Our model accurately classifies a perceived face and synthesizes the corresponding mental representation perceived by a human when observing a face in context. With this, our model offers visual insights into its internal decision-making process. We achieve this by learning two independent representations of content and context using a VAE-GAN architecture. Subsequently, we propose a novel attention mechanism for context-dependent feature adaptation. The adapted representation is used for classification and to generate a context-augmented expression. We evaluate synthesized expressions in a human study, showing that our model effectively produces approximations of human mental representations. We achieve State-of-the-Art classification accuracies of 81.01% on the RAVDESS dataset and 79.34% on the MEAD dataset. We make our code publicly available.