Learning from Heterogeneity: Generalizing Dynamic Facial Expression Recognition via Distributionally Robust Optimization

TL;DR

This paper introduces a novel distributionally robust framework for dynamic facial expression recognition that enhances model robustness and accuracy across heterogeneous data sources using attention and adaptive optimization modules.

Contribution

The paper proposes the Heterogeneity-aware Distributional Framework (HDF) with two plug-and-play modules to improve time-frequency modeling and optimize training stability for better generalization.

Findings

Significantly improves recognition accuracy on DFEW and FERV39k datasets.

Achieves superior weighted and unweighted average recall compared to existing methods.

Demonstrates robustness across diverse and imbalanced data scenarios.

Abstract

Dynamic Facial Expression Recognition (DFER) plays a critical role in affective computing and human-computer interaction. Although existing methods achieve comparable performance, they inevitably suffer from performance degradation under sample heterogeneity caused by multi-source data and individual expression variability. To address these challenges, we propose a novel framework, called Heterogeneity-aware Distributional Framework (HDF), and design two plug-and-play modules to enhance time-frequency modeling and mitigate optimization imbalance caused by hard samples. Specifically, the Time-Frequency Distributional Attention Module (DAM) captures both temporal consistency and frequency robustness through a dual-branch attention design, improving tolerance to sequence inconsistency and visual style shifts. Then, based on gradient sensitivity and information bottleneck principles, an adaptive optimization module Distribution-aware Scaling Module (DSM) is introduced to dynamically balance classification and contrastive losses, enabling more stable and discriminative representation learning. Extensive experiments on two widely used datasets, DFEW and FERV39k, demonstrate that HDF significantly improves both recognition accuracy and robustness. Our method achieves superior weighted average recall (WAR) and unweighted average recall (UAR) while maintaining strong generalization across diverse and imbalanced scenarios. Codes are released at https://github.com/QIcita/HDF_DFER.