Ordering Matters: Rank-Aware Selective Fusion for Blended Emotion Recognition

TL;DR

This paper introduces a rank-aware multi-encoder framework for blended emotion recognition that selectively fuses diverse multimodal features, improving robustness and accuracy in complex emotional expressions.

Contribution

It proposes a novel rank-aware selective fusion method with attention-based importance estimation and domain adaptation, advancing the state-of-the-art in blended emotion recognition.

Findings

Outperforms individual encoders and naive fusion baselines.

Achieved 2nd place in the BlEmoRE challenge.

Demonstrates robustness under distribution shifts.

Abstract

Blended emotion recognition is challenging because emotions are often expressed as mixtures of subtle and overlapping multimodal cues rather than a single dominant signal. We propose a rank-aware multi-encoder framework that selectively combines complementary representations from diverse pre-extracted video and audio encoders. Our method projects heterogeneous encoder features into a shared latent space, estimates sample-wise encoder importance through an attention-based gating module, and fuses only the top-n most informative encoders. To better model blended emotions, we decouple prediction into presence and salience heads and align them through probability-level fusion. We further incorporate feature-level unsupervised domain adaptation without pseudo-labeling to improve robustness under distribution shift. Experiments on the BlEmoRE challenge show that the proposed framework outperforms strong individual encoders and na\"ive multi-encoder fusion baselines. Our final system ranked 2nd in the competition, supporting the effectiveness of rank-aware selective fusion for fine-grained blended emotion recognition.