Is a Picture Worth a Thousand Words? Adaptive Multimodal Fact-Checking with Visual Evidence Necessity

TL;DR

This paper introduces AMuFC, a multimodal fact-checking framework that adaptively uses visual evidence based on necessity, improving accuracy over traditional methods that use visual data indiscriminately.

Contribution

The work presents a novel adaptive multimodal fact-checking approach with a dual-model system to determine when visual evidence is necessary, enhancing verification accuracy.

Findings

Incorporating visual evidence necessity improves accuracy.

The proposed framework outperforms existing multimodal fact-checking methods.

Code and datasets will be publicly released.

Abstract

Automated fact-checking is a crucial task that supports a responsible information ecosystem. While recent research has progressed from text-only to multimodal fact-checking, a prevailing assumption is that incorporating visual evidence universally improves performance. In this work, we challenge this assumption and show that the indiscriminate use of multimodal evidence can reduce accuracy. To address this challenge, we propose AMuFC, a multimodal fact-checking framework that employs two collaborative vision-language models with distinct roles for the adaptive use of visual evidence: an Analyzer determines whether visual evidence is necessary for claim verification, and a Verifier predicts claim veracity conditioned on both the retrieved evidence and the Analyzer's assessment. Experimental results on three datasets show that incorporating the Analyzer's assessment of visual evidence necessity into the Verifier's prediction yields substantial improvements in verification performance. We will release all code and datasets at https://github.com/ssu-humane/AMuFC.