MMM-Fact: A Multimodal, Multi-Domain Fact-Checking Dataset with Multi-Level Retrieval Difficulty

TL;DR

MMM-Fact is a comprehensive, multimodal fact-checking dataset spanning multiple domains and evidence types, designed to challenge models with varying retrieval difficulties and support advanced fact-checking tasks.

Contribution

The paper introduces MMM-Fact, a large-scale, multimodal fact-checking dataset with multi-level retrieval difficulty, covering multiple domains and evidence types, enabling more realistic fact-checking research.

Findings

Baseline models perform worse as evidence complexity increases.

MMM-Fact is significantly more challenging than previous datasets.

The dataset supports diverse fact-checking tasks including veracity prediction and evidence aggregation.

Abstract

Misinformation and disinformation demand fact checking that goes beyond simple evidence-based reasoning. Existing benchmarks fall short: they are largely single modality (text-only), span short time horizons, use shallow evidence, cover domains unevenly, and often omit full articles -- obscuring models' real-world capability. We present MMM-Fact, a large-scale benchmark of 125,449 fact-checked statements (1995--2025) across multiple domains, each paired with the full fact-check article and multimodal evidence (text, images, videos, tables) from four fact-checking sites and one news outlet. To reflect verification effort, each statement is tagged with a retrieval-difficulty tier -- Basic (1--5 sources), Intermediate (6--10), and Advanced (>10) -- supporting fairness-aware evaluation for multi-step, cross-modal reasoning. The dataset adopts a three-class veracity scheme (true/false/not enough information) and enables tasks in veracity prediction, explainable fact-checking, complex evidence aggregation, and longitudinal analysis. Baselines with mainstream LLMs show MMM-Fact is markedly harder than prior resources, with performance degrading as evidence complexity rises. MMM-Fact offers a realistic, scalable benchmark for transparent, reliable, multimodal fact-checking.