Multimodal Latent Emotion Recognition from Micro-expression and Physiological Signals

TL;DR

This paper introduces a novel multimodal learning framework that combines micro-expression and physiological signals for improved latent emotion recognition, utilizing advanced neural network components and feature fusion techniques.

Contribution

The paper presents a new multimodal framework integrating ME and PS with depthwise inception networks and guided attention modules for superior emotion recognition accuracy.

Findings

Outperforms benchmark methods in emotion recognition accuracy

Weighted feature fusion improves model performance

Attention modules enhance multimodal learning effectiveness

Abstract

This paper discusses the benefits of incorporating multimodal data for improving latent emotion recognition accuracy, focusing on micro-expression (ME) and physiological signals (PS). The proposed approach presents a novel multimodal learning framework that combines ME and PS, including a 1D separable and mixable depthwise inception network, a standardised normal distribution weighted feature fusion method, and depth/physiology guided attention modules for multimodal learning. Experimental results show that the proposed approach outperforms the benchmark method, with the weighted fusion method and guided attention modules both contributing to enhanced performance.