HEU Emotion: A Large-scale Database for Multi-modal Emotion Recognition in the Wild

TL;DR

This paper introduces HEU Emotion, a large-scale, multi-modal database for emotion recognition in natural settings, and evaluates various methods to establish benchmarks and improve multi-modal fusion techniques.

Contribution

The paper presents HEU Emotion, the largest multi-modal emotion database in the wild, and proposes a Multi-modal Attention module for improved emotion recognition accuracy.

Findings

Multi-modal fusion improves recognition accuracy by over 2% and 4%.

HEU Emotion contains nearly 20,000 videos from diverse sources.

Evaluation of machine learning and deep learning methods establishes baseline benchmarks.

Abstract

The study of affective computing in the wild setting is underpinned by databases. Existing multimodal emotion databases in the real-world conditions are few and small, with a limited number of subjects and expressed in a single language. To meet this requirement, we collected, annotated, and prepared to release a new natural state video database (called HEU Emotion). HEU Emotion contains a total of 19,004 video clips, which is divided into two parts according to the data source. The first part contains videos downloaded from Tumblr, Google, and Giphy, including 10 emotions and two modalities (facial expression and body posture). The second part includes corpus taken manually from movies, TV series, and variety shows, consisting of 10 emotions and three modalities (facial expression, body posture, and emotional speech). HEU Emotion is by far the most extensive multi-modal emotional database with 9,951 subjects. In order to provide a benchmark for emotion recognition, we used many conventional machine learning and deep learning methods to evaluate HEU Emotion. We proposed a Multi-modal Attention module to fuse multi-modal features adaptively. After multi-modal fusion, the recognition accuracies for the two parts increased by 2.19% and 4.01% respectively over those of single-modal facial expression recognition.