Conditional Information Bottleneck for Multimodal Fusion: Overcoming Shortcut Learning in Sarcasm Detection

TL;DR

This paper introduces a Multimodal Conditional Information Bottleneck model to improve sarcasm detection by reducing shortcut learning and enhancing modality fusion, leading to better generalization in real-world scenarios.

Contribution

It proposes a novel MCIB model that effectively filters out shortcut signals and improves multimodal fusion for sarcasm detection, addressing limitations of existing methods.

Findings

MCIB outperforms previous models in sarcasm detection accuracy.

Removing shortcut signals enhances model generalization.

Systematic experiments reveal weaknesses in current fusion strategies.

Abstract

Multimodal sarcasm detection is a complex task that requires distinguishing subtle complementary signals across modalities while filtering out irrelevant information. Many advanced methods rely on learning shortcuts from datasets rather than extracting intended sarcasm-related features. However, our experiments show that shortcut learning impairs the model's generalization in real-world scenarios. Furthermore, we reveal the weaknesses of current modality fusion strategies for multimodal sarcasm detection through systematic experiments, highlighting the necessity of focusing on effective modality fusion for complex emotion recognition. To address these challenges, we construct MUStARD++$^{R}$ by removing shortcut signals from MUStARD++. Then, a Multimodal Conditional Information Bottleneck (MCIB) model is introduced to enable efficient multimodal fusion for sarcasm detection. Experimental results show that the MCIB achieves the best performance without relying on shortcut learning.