A Unified Transformer-based Network for multimodal Emotion Recognition

TL;DR

This paper introduces a novel transformer-based model that combines biosensor signals and facial images to improve emotion recognition accuracy in multimodal data, demonstrating competitive results on benchmark datasets.

Contribution

The paper presents a unified transformer architecture that effectively integrates ECG/PPG signals and facial expressions for emotion recognition, a novel approach in biosensing research.

Findings

Achieved comparable results to state-of-the-art methods on MAHNOB-HCI and DEAP datasets.

Demonstrated the effectiveness of combining biosensor and facial data using a transformer model.

Validated the model's ability to learn from multimodal emotion data with minimal modality-specific design.

Abstract

The development of transformer-based models has resulted in significant advances in addressing various vision and NLP-based research challenges. However, the progress made in transformer-based methods has not been effectively applied to biosensing research. This paper presents a novel Unified Biosensor-Vision Multi-modal Transformer-based (UBVMT) method to classify emotions in an arousal-valence space by combining a 2D representation of an ECG/PPG signal with the face information. To achieve this goal, we first investigate and compare the unimodal emotion recognition performance of three image-based representations of the ECG/PPG signal. We then present our UBVMT network which is trained to perform emotion recognition by combining the 2D image-based representation of the ECG/PPG signal and the facial expression features. Our unified transformer model consists of homogeneous transformer blocks that take as an input the 2D representation of the ECG/PPG signal and the corresponding face frame for emotion representation learning with minimal modality-specific design. Our UBVMT model is trained by reconstructing masked patches of video frames and 2D images of ECG/PPG signals, and contrastive modeling to align face and ECG/PPG data. Extensive experiments on the MAHNOB-HCI and DEAP datasets show that our Unified UBVMT-based model produces comparable results to the state-of-the-art techniques.