Counterfactual Reasoning for Out-of-distribution Multimodal Sentiment Analysis

TL;DR

This paper introduces a causal inference-based counterfactual framework to improve out-of-distribution multimodal sentiment analysis by mitigating the influence of spurious textual correlations, enhancing model generalization.

Contribution

It proposes a novel counterfactual approach that separates direct and indirect effects of textual modality, addressing spurious correlations for better OOD sentiment analysis.

Findings

Outperforms existing methods in OOD sentiment tasks

Improves generalization across diverse datasets

Effectively isolates textual bias in multimodal models

Abstract

Existing studies on multimodal sentiment analysis heavily rely on textual modality and unavoidably induce the spurious correlations between textual words and sentiment labels. This greatly hinders the model generalization ability. To address this problem, we define the task of out-of-distribution (OOD) multimodal sentiment analysis. This task aims to estimate and mitigate the bad effect of textual modality for strong OOD generalization. To this end, we embrace causal inference, which inspects the causal relationships via a causal graph. From the graph, we find that the spurious correlations are attributed to the direct effect of textual modality on the model prediction while the indirect one is more reliable by considering multimodal semantics. Inspired by this, we devise a model-agnostic counterfactual framework for multimodal sentiment analysis, which captures the direct effect of textual modality via an extra text model and estimates the indirect one by a multimodal model. During the inference, we first estimate the direct effect by the counterfactual inference, and then subtract it from the total effect of all modalities to obtain the indirect effect for reliable prediction. Extensive experiments show the superior effectiveness and generalization ability of our proposed framework.