Unveiling Modality Bias: Automated Sample-Specific Analysis for Multimodal Misinformation Benchmarks

TL;DR

This paper introduces automated methods to detect modality bias at the sample level in multimodal misinformation datasets, revealing insights into bias characteristics and guiding future research.

Contribution

It proposes three novel bias quantification techniques at different granularity levels for automated sample-specific bias analysis.

Findings

Ensembling multiple views improves analysis reliability.

Automated analysis is affected by detector fluctuations.

Balanced samples show higher agreement across views.

Abstract

Numerous multimodal misinformation benchmarks exhibit bias toward specific modalities, allowing detectors to make predictions based solely on one modality. While previous research has quantified bias at the dataset level or manually identified spurious correlations between modalities and labels, these approaches lack meaningful insights at the sample level and struggle to scale to the vast amount of online information. In this paper, we investigate the design for automated recognition of modality bias at the sample level. Specifically, we propose three bias quantification methods based on theories/views of different levels of granularity: 1) a coarse-grained evaluation of modality benefit; 2) a medium-grained quantification of information flow; and 3) a fine-grained causality analysis. To verify the effectiveness, we conduct a human evaluation on two popular benchmarks. Experimental results reveal three interesting findings that provide potential direction toward future research: 1)~Ensembling multiple views is crucial for reliable automated analysis; 2)~Automated analysis is prone to detector-induced fluctuations; and 3)~Different views produce a higher agreement on modality-balanced samples but diverge on biased ones.