Cross-modal Prompting for Balanced Incomplete Multi-modal Emotion Recognition

TL;DR

This paper introduces Cross-modal Prompting (ComP), a novel approach for incomplete multi-modal emotion recognition that enhances modality-specific features and balances information across modalities, especially under missing data conditions.

Contribution

The paper proposes a new ComP method with a progressive prompt generation, cross-modal knowledge propagation, and dynamic re-weighting to improve multi-modal emotion recognition with incomplete data.

Findings

Outperforms state-of-the-art methods on 4 datasets.

Effective under various missing data rates.

Enhances modality-specific feature discrimination.

Abstract

Incomplete multi-modal emotion recognition (IMER) aims at understanding human intentions and sentiments by comprehensively exploring the partially observed multi-source data. Although the multi-modal data is expected to provide more abundant information, the performance gap and modality under-optimization problem hinder effective multi-modal learning in practice, and are exacerbated in the confrontation of the missing data. To address this issue, we devise a novel Cross-modal Prompting (ComP) method, which emphasizes coherent information by enhancing modality-specific features and improves the overall recognition accuracy by boosting each modality's performance. Specifically, a progressive prompt generation module with a dynamic gradient modulator is proposed to produce concise and consistent modality semantic cues. Meanwhile, cross-modal knowledge propagation selectively amplifies the consistent information in modality features with the delivered prompts to enhance the discrimination of the modality-specific output. Additionally, a coordinator is designed to dynamically re-weight the modality outputs as a complement to the balance strategy to improve the model's efficacy. Extensive experiments on 4 datasets with 7 SOTA methods under different missing rates validate the effectiveness of our proposed method.