Hierarchical Hypercomplex Network for Multimodal Emotion Recognition

TL;DR

This paper introduces a hierarchical hypercomplex neural network that effectively captures intra- and inter-modal relations in multimodal physiological signals, significantly improving emotion recognition accuracy over existing models.

Contribution

It proposes a novel hypercomplex network architecture with hierarchical learning for multimodal emotion recognition, leveraging parameterized hypercomplex convolutions and multiplications.

Findings

Outperforms state-of-the-art models on MAHNOB-HCI dataset

Effectively captures intra-modal and inter-modal relations

Improves classification of valence and arousal from physiological signals

Abstract

Emotion recognition is relevant in various domains, ranging from healthcare to human-computer interaction. Physiological signals, being beyond voluntary control, offer reliable information for this purpose, unlike speech and facial expressions which can be controlled at will. They reflect genuine emotional responses, devoid of conscious manipulation, thereby enhancing the credibility of emotion recognition systems. Nonetheless, multimodal emotion recognition with deep learning models remains a relatively unexplored field. In this paper, we introduce a fully hypercomplex network with a hierarchical learning structure to fully capture correlations. Specifically, at the encoder level, the model learns intra-modal relations among the different channels of each input signal. Then, a hypercomplex fusion module learns inter-modal relations among the embeddings of the different modalities. The main novelty is in exploiting intra-modal relations by endowing the encoders with parameterized hypercomplex convolutions (PHCs) that thanks to hypercomplex algebra can capture inter-channel interactions within single modalities. Instead, the fusion module comprises parameterized hypercomplex multiplications (PHMs) that can model inter-modal correlations. The proposed architecture surpasses state-of-the-art models on the MAHNOB-HCI dataset for emotion recognition, specifically in classifying valence and arousal from electroencephalograms (EEGs) and peripheral physiological signals. The code of this study is available at https://github.com/ispamm/MHyEEG.