The Psychological and Physiological Part of Emotions: Multimodal Approximation for Valence Classification

TL;DR

This study explores the psychobiology of central and peripheral nervous system signals for emotion recognition, using EEG, ECG, and skin temperature data from 24 subjects, and finds multimodal approaches do not outperform EEG alone.

Contribution

It provides a computational model for valence emotion recognition based on psychobiological signals and compares the effectiveness of multimodal versus single modality approaches.

Findings

EEG alone effectively classifies valence emotions.

Multimodal approach did not improve classification over EEG alone.

Sex differences influence physiological responses to emotions.

Abstract

In order to develop more precise and functional affective applications, it is necessary to achieve a balance between the psychology and the engineering applied to emotions. Signals from the central and peripheral nervous systems have been used for emotion recognition purposes, however, their operation and the relationship between them remains unknown. In this context, in the present work we have tried to approach the study of the psychobiology of both systems in order to generate a computational model for the recognition of emotions in the dimension of valence. To this end, the electroencephalography (EEG) signal, electrocardiography (ECG) signal and skin temperature of 24 subjects have been studied. Each methodology has been evaluated individually, finding characteristic patterns of positive and negative emotions in each of them. After feature selection of each methodology, the results of the classification showed that, although the classification of emotions is possible at both central and peripheral levels, the multimodal approach did not improve the results obtained through the EEG alone. In addition, differences have been observed between cerebral and physiological responses in the processing emotions by separating the sample by sex; though, the differences between men and women were only notable at the physiological level.