Impact of multiple modalities on emotion recognition: investigation into 3d facial landmarks, action units, and physiological data

TL;DR

This paper investigates how combining 3D facial landmarks, action units, and physiological data affects emotion recognition, highlighting the strengths of facial and physiological features and the challenges of using action units alone.

Contribution

It provides a comprehensive analysis of multiple modalities for emotion recognition and evaluates their individual and combined effectiveness.

Findings

3D facial landmarks and physiological data are effective for emotion recognition.

Fusion of modalities improves recognition accuracy.

Action units alone are less effective but can enhance performance when combined with other data.

Abstract

To fully understand the complexities of human emotion, the integration of multiple physical features from different modalities can be advantageous. Considering this, we present an analysis of 3D facial data, action units, and physiological data as it relates to their impact on emotion recognition. We analyze each modality independently, as well as the fusion of each for recognizing human emotion. This analysis includes which features are most important for specific emotions (e.g. happy). Our analysis indicates that both 3D facial landmarks and physiological data are encouraging for expression/emotion recognition. On the other hand, while action units can positively impact emotion recognition when fused with other modalities, the results suggest it is difficult to detect emotion using them in a unimodal fashion.