MuVaC: A Variational Causal Framework for Multimodal Sarcasm Understanding in Dialogues

TL;DR

MuVaC is a novel variational causal inference framework that jointly models multimodal sarcasm detection and explanation, mimicking human reasoning to improve understanding of sarcasm in dialogues.

Contribution

It introduces a causal modeling approach that integrates detection and explanation tasks, addressing their dependency and enhancing multimodal sarcasm understanding.

Findings

MuVaC outperforms existing methods on public datasets.

The framework effectively models causal relationships for sarcasm analysis.

Joint optimization improves both detection accuracy and explanation quality.

Abstract

The prevalence of sarcasm in multimodal dialogues on the social platforms presents a crucial yet challenging task for understanding the true intent behind online content. Comprehensive sarcasm analysis requires two key aspects: Multimodal Sarcasm Detection (MSD) and Multimodal Sarcasm Explanation (MuSE). Intuitively, the act of detection is the result of the reasoning process that explains the sarcasm. Current research predominantly focuses on addressing either MSD or MuSE as a single task. Even though some recent work has attempted to integrate these tasks, their inherent causal dependency is often overlooked. To bridge this gap, we propose MuVaC, a variational causal inference framework that mimics human cognitive mechanisms for understanding sarcasm, enabling robust multimodal feature learning to jointly optimize MSD and MuSE. Specifically, we first model MSD and MuSE from the perspective of structural causal models, establishing variational causal pathways to define the objectives for joint optimization. Next, we design an alignment-then-fusion approach to integrate multimodal features, providing robust fusion representations for sarcasm detection and explanation generation. Finally, we enhance the reasoning trustworthiness by ensuring consistency between detection results and explanations. Experimental results demonstrate the superiority of MuVaC in public datasets, offering a new perspective for understanding multimodal sarcasm.