Hierarchical MoE: Continuous Multimodal Emotion Recognition with Incomplete and Asynchronous Inputs

TL;DR

This paper introduces Hi-MoE, a hierarchical mixture-of-experts framework that enhances continuous multimodal emotion recognition by effectively handling incomplete and asynchronous data, improving robustness and accuracy in real-world scenarios.

Contribution

The proposed Hi-MoE framework introduces a dual-layer expert structure with dynamic routing and cross-modal alignment, advancing robustness and adaptability in multimodal emotion recognition.

Findings

Achieves state-of-the-art performance on DEAP and DREAMER datasets.

Demonstrates robustness under modality incompleteness and asynchrony.

Outperforms existing methods in continuous emotion regression.

Abstract

Multimodal emotion recognition (MER) is crucial for human-computer interaction, yet real-world challenges like dynamic modality incompleteness and asynchrony severely limit its robustness. Existing methods often assume consistently complete data or lack dynamic adaptability. To address these limitations, we propose a novel Hi-MoE~(Hierarchical Mixture-of-Experts) framework for robust continuous emotion prediction. This framework employs a dual-layer expert structure. A Modality Expert Bank utilizes soft routing to dynamically handle missing modalities and achieve robust information fusion. A subsequent Emotion Expert Bank leverages differential-attention routing to flexibly attend to emotional prototypes, enabling fine-grained emotion representation. Additionally, a cross-modal alignment module explicitly addresses temporal shifts and semantic inconsistencies between modalities. Extensive experiments on benchmark datasets DEAP and DREAMER demonstrate our model's state-of-the-art performance in continuous emotion regression, showcasing exceptional robustness under challenging conditions such as dynamic modality absence and asynchronous sampling. This research significantly advances the development of intelligent emotion systems adaptable to complex real-world environments.