Multimodal Emotion Recognition Using Deep Canonical Correlation Analysis

TL;DR

This paper introduces deep canonical correlation analysis (DCCA) for multimodal emotion recognition, demonstrating its superior accuracy and robustness across five datasets by transforming and coordinating multiple modalities into a discriminative hyperspace.

Contribution

The paper proposes a novel DCCA method that effectively transforms and aligns multimodal data for improved emotion recognition performance.

Findings

DCCA achieves state-of-the-art accuracy on five datasets.

DCCA exhibits greater noise robustness than existing methods.

Features transformed by DCCA are more homogeneous and discriminative.

Abstract

Multimodal signals are more powerful than unimodal data for emotion recognition since they can represent emotions more comprehensively. In this paper, we introduce deep canonical correlation analysis (DCCA) to multimodal emotion recognition. The basic idea behind DCCA is to transform each modality separately and coordinate different modalities into a hyperspace by using specified canonical correlation analysis constraints. We evaluate the performance of DCCA on five multimodal datasets: the SEED, SEED-IV, SEED-V, DEAP, and DREAMER datasets. Our experimental results demonstrate that DCCA achieves state-of-the-art recognition accuracy rates on all five datasets: 94.58% on the SEED dataset, 87.45% on the SEED-IV dataset, 84.33% and 85.62% for two binary classification tasks and 88.51% for a four-category classification task on the DEAP dataset, 83.08% on the SEED-V dataset, and 88.99%, 90.57%, and 90.67% for three binary classification tasks on the DREAMER dataset. We also compare the noise robustness of DCCA with that of existing methods when adding various amounts of noise to the SEED-V dataset. The experimental results indicate that DCCA has greater robustness. By visualizing feature distributions with t-SNE and calculating the mutual information between different modalities before and after using DCCA, we find that the features transformed by DCCA from different modalities are more homogeneous and discriminative across emotions.