Mitigating GenAI-powered Evidence Pollution for Out-of-Context Multimodal Misinformation Detection

TL;DR

This paper investigates how GenAI-generated evidence pollution impacts multimodal misinformation detection and proposes strategies to improve robustness against such pollution.

Contribution

It introduces two novel strategies, cross-modal evidence reranking and reasoning, to mitigate evidence pollution effects in out-of-context misinformation detection.

Findings

Polluted evidence degrades detection performance by over 9 percentage points.

Proposed strategies significantly improve robustness against evidence pollution.

Experiments on benchmark datasets validate the effectiveness of the methods.

Abstract

While large generative artificial intelligence (GenAI) models have achieved significant success, they also raise growing concerns about online information security due to their potential misuse for generating deceptive content. Out-of-context (OOC) multimodal misinformation detection, which often retrieves Web evidence to identify the repurposing of images in false contexts, faces the issue of reasoning over GenAI-polluted evidence to derive accurate predictions. Existing works simulate GenAI-powered pollution at the claim level with stylistic rewriting to conceal linguistic cues, and ignore evidence-level pollution for such information-seeking applications. In this work, we investigate how polluted evidence affects the performance of existing OOC detectors, revealing a performance degradation of more than 9 percentage points. We propose two strategies, cross-modal evidence reranking and cross-modal claim-evidence reasoning, to address the challenges posed by polluted evidence. Extensive experiments on two benchmark datasets show that these strategies can effectively enhance the robustness of existing out-of-context detectors amidst polluted evidence.