MAD: A Multimodal and Multi-perspective Affective Dataset with Hierarchical Annotations

TL;DR

MAD is a comprehensive multimodal dataset capturing neural, physiological, and facial data with hierarchical emotion annotations, enabling advanced affective computing research and cross-modal emotion analysis.

Contribution

It introduces a novel multimodal dataset with synchronized neural, physiological, and facial data, along with a hierarchical annotation framework for emotion research.

Findings

Supports consistent emotion recognition across modalities

Enables cross-subject transfer learning experiments

Provides a reliable benchmark for affective computing

Abstract

This work presents MAD (Multimodal Affection Dataset), a multimodal emotion dataset designed for affective computing and neurophysiological modeling. MAD is built upon synchronous collection of diverse physiological signals (EEG, ECG, EOG, EMG, PPG, and BCG) together with tri-view RGB-D facial videos, enabling the observation of emotional dynamics from neural, physiological, and behavioral perspectives. The dataset consists of synchronized recordings from 18 participants and introduces two key contributions. First, it provides temporally aligned multimodal data that jointly capture central neural activity, peripheral physiological responses, and overt facial expressions. Second, it incorporates a three-level emotion annotation framework spanning stimulus elicitation, subjective cognition, and behavioral expression, supporting joint modeling of the full emotion process. To validate the dataset, we conduct systematic benchmark experiments covering intra-subject EEG emotion recognition, cross-subject EEG transfer learning, consistency analysis and emotion classification with cardiac-related signals, multimodal physiological fusion, and multi-view facial emotion recognition. The experimental results demonstrate that MAD supports consistent and comparable performance across both unimodal and multimodal settings, establishing it as a reliable benchmark for emotion recognition and cross-modal affective analysis, and as a valuable resource for studying emotion mechanisms across multiple levels.