Attentive Cross-modal Connections for Deep Multimodal Wearable-based Emotion Recognition

TL;DR

This paper introduces an attentive cross-modal connection mechanism that enhances deep multimodal emotion recognition by sharing and weighting intermediate representations between ECG and EDA signals, leading to improved classification performance.

Contribution

It proposes a novel attentive cross-modal connection for deep neural networks that effectively shares and weights intermediate features between modalities for emotion recognition.

Findings

The method outperforms baseline models on the WESAD dataset.

Attention-weighted sharing improves multimodal embedding quality.

The approach effectively combines ECG and EDA signals for emotion classification.

Abstract

Classification of human emotions can play an essential role in the design and improvement of human-machine systems. While individual biological signals such as Electrocardiogram (ECG) and Electrodermal Activity (EDA) have been widely used for emotion recognition with machine learning methods, multimodal approaches generally fuse extracted features or final classification/regression results to boost performance. To enhance multimodal learning, we present a novel attentive cross-modal connection to share information between convolutional neural networks responsible for learning individual modalities. Specifically, these connections improve emotion classification by sharing intermediate representations among EDA and ECG and apply attention weights to the shared information, thus learning more effective multimodal embeddings. We perform experiments on the WESAD dataset to identify the best configuration of the proposed method for emotion classification. Our experiments show that the proposed approach is capable of learning strong multimodal representations and outperforms a number of baselines methods.