Towards Robust Multimodal Emotion Recognition under Missing Modalities and Distribution Shifts

TL;DR

This paper introduces CIDer, a robust multimodal emotion recognition framework that handles missing modalities and distribution shifts using causal inference and self-distillation, improving performance and efficiency.

Contribution

The paper proposes CIDer, a novel MER framework combining causal inference and self-distillation, and introduces RMFM and new OOD datasets for better generalization under challenging conditions.

Findings

CIDer outperforms state-of-the-art methods in RMFM and OOD scenarios.

CIDer achieves comparable or better accuracy with fewer parameters.

The new datasets facilitate evaluation of robustness in MER models.

Abstract

Recent advancements in Multimodal Emotion Recognition (MER) face challenges in addressing both modality missing and Out-Of-Distribution (OOD) data simultaneously. Existing methods often rely on specific models or introduce excessive parameters, which limits their practicality. To address these issues, we propose a novel robust MER framework, Causal Inference Distiller (CIDer), and introduce a new task, Random Modality Feature Missing (RMFM), to generalize the definition of modality missing. CIDer integrates two key components: a Model-Specific Self-Distillation (MSSD) module and a Model-Agnostic Causal Inference (MACI) module. MSSD enhances robustness under the RMFM task through a weight-sharing self-distillation approach applied across low-level features, attention maps, and high-level representations. Additionally, a Word-level Self-aligned Attention Module (WSAM) reduces computational complexity, while a Multimodal Composite Transformer (MCT) facilitates efficient multimodal fusion. To tackle OOD challenges, MACI employs a tailored causal graph to mitigate label and language biases using a Multimodal Causal Module (MCM) and fine-grained counterfactual texts. Notably, MACI can independently enhance OOD generalization with minimal additional parameters. Furthermore, we also introduce the new repartitioned MER OOD datasets. Experimental results demonstrate that CIDer achieves robust performance in both RMFM and OOD scenarios, with fewer parameters and faster training compared to state-of-the-art methods. The implementation of this work is publicly accessible at https://github.com/gw-zhong/CIDer.