Weak Supervision with Arbitrary Single Frame for Micro- and Macro-expression Spotting

TL;DR

This paper introduces a weakly-supervised framework for micro- and macro-expression spotting that requires only a single annotated frame per expression, significantly reducing annotation effort while maintaining high accuracy.

Contribution

The proposed PWES framework employs novel pseudo-labeling and contrastive learning strategies to improve localization accuracy with minimal supervision.

Findings

Achieves performance comparable to fully-supervised methods

Effective pseudo-label generation improves localization accuracy

Global feature representation enhances model robustness

Abstract

Frame-level micro- and macro-expression spotting methods require time-consuming frame-by-frame observation during annotation. Meanwhile, video-level spotting lacks sufficient information about the location and number of expressions during training, resulting in significantly inferior performance compared with fully-supervised spotting. To bridge this gap, we propose a point-level weakly-supervised expression spotting (PWES) framework, where each expression requires to be annotated with only one random frame (i.e., a point). To mitigate the issue of sparse label distribution, the prevailing solution is pseudo-label mining, which, however, introduces new problems: localizing contextual background snippets results in inaccurate boundaries and discarding foreground snippets leads to fragmentary predictions. Therefore, we design the strategies of multi-refined pseudo label generation (MPLG) and distribution-guided feature contrastive learning (DFCL) to address these problems. Specifically, MPLG generates more reliable pseudo labels by merging class-specific probabilities, attention scores, fused features, and point-level labels. DFCL is utilized to enhance feature similarity for the same categories and feature variability for different categories while capturing global representations across the entire datasets. Extensive experiments on the CAS(ME)^2, CAS(ME)^3, and SAMM-LV datasets demonstrate PWES achieves promising performance comparable to that of recent fully-supervised methods.