Multimodal Emotion Recognition Using Multimodal Deep Learning

TL;DR

This paper presents a multimodal deep learning framework for emotion recognition using physiological signals, achieving high accuracy and demonstrating the effectiveness of shared representations across modalities.

Contribution

It introduces Bimodal Deep AutoEncoder models that significantly improve emotion recognition accuracy over existing methods.

Findings

Achieved 82.11% accuracy on SEED with DAE

Reached 91.01% and 83.25% accuracy with BDAE on SEED and DEAP datasets

Demonstrated cross-modal learning with 66.34% accuracy using shared representations

Abstract

To enhance the performance of affective models and reduce the cost of acquiring physiological signals for real-world applications, we adopt multimodal deep learning approach to construct affective models from multiple physiological signals. For unimodal enhancement task, we indicate that the best recognition accuracy of 82.11% on SEED dataset is achieved with shared representations generated by Deep AutoEncoder (DAE) model. For multimodal facilitation tasks, we demonstrate that the Bimodal Deep AutoEncoder (BDAE) achieves the mean accuracies of 91.01% and 83.25% on SEED and DEAP datasets, respectively, which are much superior to the state-of-the-art approaches. For cross-modal learning task, our experimental results demonstrate that the mean accuracy of 66.34% is achieved on SEED dataset through shared representations generated by EEG-based DAE as training samples and shared representations generated by eye-based DAE as testing sample, and vice versa.