Temporal and Spatial Feature Fusion Framework for Dynamic Micro Expression Recognition

TL;DR

This paper introduces a novel framework combining temporal and spatial feature fusion with advanced neural networks to improve the accuracy of dynamic micro-expression recognition, addressing the challenge of recognizing highly transient facial cues.

Contribution

The paper proposes a new Temporal and Spatial feature Fusion framework (TSFmicro) that integrates RetNet and transformer networks with a novel parallel time-space fusion method for better micro-expression recognition.

Findings

Outperforms state-of-the-art methods on three micro-expression datasets.

Effectively captures and fuses spatio-temporal features for improved accuracy.

Demonstrates robustness and efficiency in recognizing micro-expressions.

Abstract

When emotions are repressed, an individual's true feelings may be revealed through micro-expressions. Consequently, micro-expressions are regarded as a genuine source of insight into an individual's authentic emotions. However, the transient and highly localised nature of micro-expressions poses a significant challenge to their accurate recognition, with the accuracy rate of micro-expression recognition being as low as 50%, even for professionals. In order to address these challenges, it is necessary to explore the field of dynamic micro expression recognition (DMER) using multimodal fusion techniques, with special attention to the diverse fusion of temporal and spatial modal features. In this paper, we propose a novel Temporal and Spatial feature Fusion framework for DMER (TSFmicro). This framework integrates a Retention Network (RetNet) and a transformer-based DMER network, with the objective of efficient micro-expression recognition through the capture and fusion of temporal and spatial relations. Meanwhile, we propose a novel parallel time-space fusion method from the perspective of modal fusion, which fuses spatio-temporal information in high-dimensional feature space, resulting in complementary "where-how" relationships at the semantic level and providing richer semantic information for the model. The experimental results demonstrate the superior performance of the TSFmicro method in comparison to other contemporary state-of-the-art methods. This is evidenced by its effectiveness on three well-recognised micro-expression datasets.