Beyond Retrieval: Improving Evidence Quality for LLM-based Multimodal Fact-Checking

TL;DR

This paper introduces Aletheia, an end-to-end multimodal fact-checking framework that enhances evidence retrieval quality, leading to significant improvements in verifying disinformation using LLMs.

Contribution

The paper presents a novel evidence retrieval strategy within Aletheia that improves coverage and filters irrelevant information, advancing multimodal fact-checking accuracy.

Findings

Achieves 88.3% accuracy on public datasets

Improves verification accuracy by up to 30.8%

Enhances evidence quality for LLM-based disinformation detection

Abstract

The increasing multimodal disinformation, where deceptive claims are reinforced through coordinated text and visual content, poses significant challenges to automated fact-checking. Recent efforts leverage Large Language Models (LLMs) for this task, capitalizing on their strong reasoning and multimodal understanding capabilities. Emerging retrieval-augmented frameworks further equip LLMs with access to open-domain external information, enabling evidence-based verification beyond their internal knowledge. Despite their promising gains, our empirical study reveals notable shortcomings in the external search coverage and evidence quality evaluation. To mitigate those limitations, we propose Aletheia, an end-to-end framework for automated multimodal fact-checking. It introduces a novel evidence retrieval strategy that improves evidence coverage and filters useless information from open-domain sources, enabling the extraction of high-quality evidence for verification. Extensive experiments demonstrate that Aletheia achieves an accuracy of 88.3% on two public multimodal disinformation datasets and 90.2% on newly emerging claims. Compared with existing evidence retrieval strategies, our approach improves verification accuracy by up to 30.8%, highlighting the critical role of evidence quality in LLM-based disinformation verification.