Towards Multimodal Sentiment Analysis Debiasing via Bias Purification

TL;DR

This paper introduces a causality-based framework called MCIS to identify and mitigate biases in multimodal sentiment analysis, improving model robustness and accuracy.

Contribution

It proposes a novel causality-driven bias purification method for multimodal sentiment analysis, addressing dataset biases and enhancing decision unbiasedness.

Findings

Effective bias mitigation demonstrated on standard benchmarks.

Significant performance improvements over baseline models.

Qualitative analysis confirms bias reduction.

Abstract

Multimodal Sentiment Analysis (MSA) aims to understand human intentions by integrating emotion-related clues from diverse modalities, such as visual, language, and audio. Unfortunately, the current MSA task invariably suffers from unplanned dataset biases, particularly multimodal utterance-level label bias and word-level context bias. These harmful biases potentially mislead models to focus on statistical shortcuts and spurious correlations, causing severe performance bottlenecks. To alleviate these issues, we present a Multimodal Counterfactual Inference Sentiment (MCIS) analysis framework based on causality rather than conventional likelihood. Concretely, we first formulate a causal graph to discover harmful biases from already-trained vanilla models. In the inference phase, given a factual multimodal input, MCIS imagines two counterfactual scenarios to purify and mitigate these biases. Then, MCIS can make unbiased decisions from biased observations by comparing factual and counterfactual outcomes. We conduct extensive experiments on several standard MSA benchmarks. Qualitative and quantitative results show the effectiveness of the proposed framework.