ME-TST+: Micro-expression Analysis via Temporal State Transition with ROI Relationship Awareness

TL;DR

This paper introduces ME-TST+ architectures that leverage temporal state transitions and ROI relationships for more accurate micro-expression analysis, outperforming traditional sliding-window methods.

Contribution

The paper proposes novel state space model-based architectures for joint micro-expression spotting and recognition, incorporating ROI relationship awareness and multi-granularity modeling.

Findings

Achieves state-of-the-art performance on micro-expression datasets.

Effectively models variable-duration micro-expressions with temporal state transitions.

Enhances analysis accuracy through feature and result-level synergy strategies.

Abstract

Micro-expressions (MEs) are regarded as important indicators of an individual's intrinsic emotions, preferences, and tendencies. ME analysis requires spotting of ME intervals within long video sequences and recognition of their corresponding emotional categories. Previous deep learning approaches commonly employ sliding-window classification networks. However, the use of fixed window lengths and hard classification presents notable limitations in practice. Furthermore, these methods typically treat ME spotting and recognition as two separate tasks, overlooking the essential relationship between them. To address these challenges, this paper proposes two state space model-based architectures, namely ME-TST and ME-TST+, which utilize temporal state transition mechanisms to replace conventional window-level classification with video-level regression. This enables a more precise characterization of the temporal dynamics of MEs and supports the modeling of MEs with varying durations. In ME-TST+, we further introduce multi-granularity ROI modeling and the slowfast Mamba framework to alleviate information loss associated with treating ME analysis as a time-series task. Additionally, we propose a synergy strategy for spotting and recognition at both the feature and result levels, leveraging their intrinsic connection to enhance overall analysis performance. Extensive experiments demonstrate that the proposed methods achieve state-of-the-art performance. The codes are available at https://github.com/zizheng-guo/ME-TST.