Learning to Rank Onset-Occurring-Offset Representations for Micro-Expression Recognition

TL;DR

This paper introduces LTR3O, a deep learning approach for micro-expression recognition that uses a flexible 3O sequence structure to improve feature learning and achieve superior performance across multiple datasets.

Contribution

The paper proposes a novel 3O structure for representing micro-expressions and a learning to rank framework that enhances discriminative feature extraction without requiring precise frame spotting.

Findings

LTR3O outperforms state-of-the-art MER methods

The 3O structure is flexible and does not need accurate frame spotting

Experimental results on three datasets confirm the effectiveness of LTR3O

Abstract

This paper focuses on the research of micro-expression recognition (MER) and proposes a flexible and reliable deep learning method called learning to rank onset-occurring-offset representations (LTR3O). The LTR3O method introduces a dynamic and reduced-size sequence structure known as 3O, which consists of onset, occurring, and offset frames, for representing micro-expressions (MEs). This structure facilitates the subsequent learning of ME-discriminative features. A noteworthy advantage of the 3O structure is its flexibility, as the occurring frame is randomly extracted from the original ME sequence without the need for accurate frame spotting methods. Based on the 3O structures, LTR3O generates multiple 3O representation candidates for each ME sample and incorporates well-designed modules to measure and calibrate their emotional expressiveness. This calibration process ensures that the distribution of these candidates aligns with that of macro-expressions (MaMs) over time. Consequently, the visibility of MEs can be implicitly enhanced, facilitating the reliable learning of more discriminative features for MER. Extensive experiments were conducted to evaluate the performance of LTR3O using three widely-used ME databases: CASME II, SMIC, and SAMM. The experimental results demonstrate the effectiveness and superior performance of LTR3O, particularly in terms of its flexibility and reliability, when compared to recent state-of-the-art MER methods.