Shallow Optical Flow Three-Stream CNN for Macro- and Micro-Expression Spotting from Long Videos

TL;DR

This paper introduces SOFTNet, a shallow three-stream CNN that effectively detects micro- and macro-expressions in long videos by framing the task as a regression problem with pseudo-labeling, achieving state-of-the-art results.

Contribution

The paper presents a novel shallow optical flow three-stream CNN model for expression spotting, utilizing regression and pseudo-labeling to improve accuracy in long videos.

Findings

Achieves state-of-the-art performance on CAS(ME)^2 benchmark.

Demonstrates effectiveness on SAMM Long Videos.

Efficient in spotting micro- and macro-expressions.

Abstract

Facial expressions vary from the visible to the subtle. In recent years, the analysis of micro-expressions $-$ a natural occurrence resulting from the suppression of one's true emotions, has drawn the attention of researchers with a broad range of potential applications. However, spotting microexpressions in long videos becomes increasingly challenging when intertwined with normal or macro-expressions. In this paper, we propose a shallow optical flow three-stream CNN (SOFTNet) model to predict a score that captures the likelihood of a frame being in an expression interval. By fashioning the spotting task as a regression problem, we introduce pseudo-labeling to facilitate the learning process. We demonstrate the efficacy and efficiency of the proposed approach on the recent MEGC 2020 benchmark, where state-of-the-art performance is achieved on CAS(ME)$^{2}$ with equally promising results on SAMM Long Videos.