Emotion Recognition from Multiple Modalities: Fundamentals and Methodologies

TL;DR

This paper provides a comprehensive overview of multi-modal emotion recognition, covering fundamental models, methodologies, challenges, and applications in enabling machines to interpret human emotions across various modalities.

Contribution

It offers a systematic summary of existing approaches, challenges, and future directions in multi-modal emotion recognition, serving as a foundational tutorial in the field.

Findings

Summarizes key emotion representation models and affective modalities.

Reviews strategies for emotion annotation and computational tasks.

Discusses representation learning, feature fusion, and domain adaptation techniques.

Abstract

Humans are emotional creatures. Multiple modalities are often involved when we express emotions, whether we do so explicitly (e.g., facial expression, speech) or implicitly (e.g., text, image). Enabling machines to have emotional intelligence, i.e., recognizing, interpreting, processing, and simulating emotions, is becoming increasingly important. In this tutorial, we discuss several key aspects of multi-modal emotion recognition (MER). We begin with a brief introduction on widely used emotion representation models and affective modalities. We then summarize existing emotion annotation strategies and corresponding computational tasks, followed by the description of main challenges in MER. Furthermore, we present some representative approaches on representation learning of each affective modality, feature fusion of different affective modalities, classifier optimization for MER, and domain adaptation for MER. Finally, we outline several real-world applications and discuss some future directions.