LEAF: Unveiling Two Sides of the Same Coin in Semi-supervised Facial Expression Recognition

TL;DR

LEAF is a semi-supervised facial expression recognition framework that simultaneously enhances expression-relevant representations and pseudo-label quality through hierarchical aggregation and decoupling strategies.

Contribution

The paper introduces LEAF, a novel hierarchical framework that decouples and fuses expression-relevant features and pseudo-labels for improved semi-supervised FER.

Findings

LEAF outperforms state-of-the-art methods on benchmark datasets.

The hierarchical aggregation improves representation quality.

Decoupling and fusing strategies enhance pseudo-label accuracy.

Abstract

Semi-supervised learning has emerged as a promising approach to tackle the challenge of label scarcity in facial expression recognition (FER) task. However, current state-of-the-art methods primarily focus on one side of the coin, i.e., generating high-quality pseudo-labels, while overlooking the other side: enhancing expression-relevant representations. In this paper, we unveil both sides of the coin by proposing a unified framework termed hierarchicaL dEcoupling And Fusing (LEAF) to coordinate expression-relevant representations and pseudo-labels for semi-supervised FER. LEAF introduces a hierarchical expression-aware aggregation strategy that operates at three levels: semantic, instance, and category. (1) At the semantic and instance levels, LEAF decouples representations into expression-agnostic and expression-relevant components, and adaptively fuses them using learnable gating weights. (2) At the category level, LEAF assigns ambiguous pseudo-labels by decoupling predictions into positive and negative parts, and employs a consistency loss to ensure agreement between two augmented views of the same image. Extensive experiments on benchmark datasets demonstrate that by unveiling and harmonizing both sides of the coin, LEAF outperforms state-of-the-art semi-supervised FER methods, effectively leveraging both labeled and unlabeled data. Moreover, the proposed expression-aware aggregation strategy can be seamlessly integrated into existing semi-supervised frameworks, leading to significant performance gains. Our code is available at \url{https://github.com/zfkarl/LEAF}.