DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Experts

TL;DR

DEFAME is a modular, zero-shot multimodal fact-checking system that dynamically verifies text and images in claims, outperforming previous methods and setting new benchmarks for open-domain fact verification.

Contribution

It introduces DEFAME, a novel end-to-end multimodal fact-checking pipeline that dynamically selects evidence and search strategies, and establishes a new benchmark ClaimReview2024+.

Findings

DEFAME surpasses previous state-of-the-art methods on VERITE, AVerITeC, and MOCHEG.

It outperforms GPT-4o baselines on ClaimReview2024+ for claims beyond GPT-4o's knowledge cutoff.

The system demonstrates strong temporal generalizability and potential for real-time fact-checking.

Abstract

The proliferation of disinformation demands reliable and scalable fact-checking solutions. We present Dynamic Evidence-based FAct-checking with Multimodal Experts (DEFAME), a modular, zero-shot MLLM pipeline for open-domain, text-image claim verification. DEFAME operates in a six-stage process, dynamically selecting the tools and search depth to extract and evaluate textual and visual evidence. Unlike prior approaches that are text-only, lack explainability, or rely solely on parametric knowledge, DEFAME performs end-to-end verification, accounting for images in claims and evidence while generating structured, multimodal reports. Evaluation on the popular benchmarks VERITE, AVerITeC, and MOCHEG shows that DEFAME surpasses all previous methods, establishing itself as the new state-of-the-art fact-checking system for uni- and multimodal fact-checking. Moreover, we introduce a new multimodal benchmark, ClaimReview2024+, featuring claims after the knowledge cutoff of GPT-4o, avoiding data leakage. Here, DEFAME drastically outperforms the GPT-4o baselines, showing temporal generalizability and the potential for real-time fact-checking.