Machine Learning-Based Ear Thermal Imaging for Emotion Sensing

TL;DR

This study shows that using machine learning on ear thermal images can better detect emotions than traditional methods.

Contribution

The study introduces deep learning for analyzing ear thermal imaging to capture nonlinear emotional responses.

Findings

Nonlinear machine learning models outperformed linear regression in predicting arousal ratings from ear thermal data.

ResNet-50 performed better than random forest in modeling the relationship between ear temperature and emotion.

Specific ear regions showed nonlinear associations with subjective arousal when analyzed using deep learning.

Abstract

Thermal imaging, which is contact-free, light-independent, and effective in detecting skin temperature changes that reflect autonomic nervous system activity, is expected to be useful for emotion sensing. A recent thermography study demonstrated a linear relationship between ear temperatures and emotional arousal ratings. However, whether and how ear thermal changes may be nonlinearly related to subjective emotions remains untested. To address this issue, we reanalyzed a dataset that included ear thermal images and self-reported arousal ratings obtained while participants watched emotion-eliciting films. We employed linear regression and two nonlinear machine learning models: a random forest model and a ResNet-50 convolutional neural network. Model evaluation using mean squared error and correlation coefficients between actual arousal ratings and model predictions indicated that both machine learning models outperformed linear regression and that the ResNet-50 model outperformed the random forest model. Interpretation of the ResNet-50 model using Gradient-weighted Class Activation Mapping and Shapley additive explanation methods revealed nonlinear associations between temperature changes in specific ear regions and subjective arousal ratings. These findings imply that ear thermal imaging combined with machine learning, particularly deep learning, holds promise for emotion sensing.

Figures