HADUA: Hierarchical Attention and Dynamic Uniform Alignment for Robust Cross-Subject Emotion Recognition

TL;DR

HADUA introduces a hierarchical attention and domain adaptation framework that improves cross-subject emotion recognition from multimodal signals by modeling intra- and inter-modal dynamics and handling pseudo-label noise.

Contribution

The paper presents a novel adaptive learning framework combining hierarchical attention, confidence-aware weighting, and uniform alignment for robust cross-subject emotion recognition.

Findings

HADUA outperforms state-of-the-art methods in accuracy and robustness.

The hierarchical attention module effectively models intra- and inter-modal interactions.

The confidence-aware scheme reduces the impact of noisy pseudo-labels.

Abstract

Robust cross-subject emotion recognition from multimodal physiological signals remains a challenging problem, primarily due to modality heterogeneity and inter-subject distribution shift. To tackle these challenges, we propose a novel adaptive learning framework named Hierarchical Attention and Dynamic Uniform Alignment (HADUA). Our approach unifies the learning of multimodal representations with domain adaptation. First, we design a hierarchical attention module that explicitly models intra-modal temporal dynamics and inter-modal semantic interactions (e.g., between electroencephalogram(EEG) and eye movement(EM)), yielding discriminative and semantically coherent fused features. Second, to overcome the noise inherent in pseudo-labels during adaptation, we introduce a confidence-aware Gaussian weighting scheme that smooths the supervision from target-domain samples by down-weighting uncertain instances. Third, a uniform alignment loss is employed to regularize the distribution of pseudo-labels across classes, thereby mitigating imbalance and stabilizing conditional distribution matching. Extensive experiments on multiple cross-subject emotion recognition benchmarks show that HADUA consistently surpasses existing state-of-the-art methods in both accuracy and robustness, validating its effectiveness in handling modality gaps, noisy pseudo-labels, and class imbalance. Taken together, these contributions offer a practical and generalizable solution for building robust cross-subject affective computing systems.