VeriTaS: The First Dynamic Benchmark for Multimodal Automated Fact-Checking

TL;DR

VeriTaS is a novel dynamic, multimodal benchmark for automated fact-checking that adapts over time to prevent data leakage and better evaluate models in real-world scenarios.

Contribution

It introduces the first dynamic, multilingual, multimodal AFC benchmark with automated claim updating and a standardized scoring scheme, addressing limitations of static benchmarks.

Findings

Automated annotations closely match human judgments.

VeriTaS covers 54 languages and includes textual and audiovisual claims.

The benchmark is designed to be leakage-resistant and adaptable over time.

Abstract

The growing scale of online misinformation urgently demands Automated Fact-Checking (AFC). Existing benchmarks for evaluating AFC systems, however, are largely limited in terms of task scope, modalities, domain, language diversity, realism, or coverage of misinformation types. Critically, they are static, thus subject to data leakage as their claims enter the pretraining corpora of LLMs. As a result, benchmark performance no longer reliably reflects the actual ability to verify claims. We introduce Verified Theses and Statements (VeriTaS), the first dynamic benchmark for multimodal AFC, designed to remain robust under ongoing large-scale pretraining of foundation models. VeriTaS currently comprises 25,000 real-world claims from 104 professional fact-checking organizations across 54 languages, covering textual and audiovisual content. Claims are added quarterly via a fully automated seven-stage pipeline that normalizes claim formulation, retrieves original media, and maps heterogeneous expert verdicts to a novel, standardized, and disentangled scoring scheme with textual justifications. Through human evaluation, we demonstrate that the automated annotations closely match human judgments. We commit to updating VeriTaS in the future, establishing a leakage-resistant benchmark, supporting meaningful AFC evaluation in the era of rapidly evolving foundation models. The code and data are publicly available under https://veritas.mai.informatik.tu-darmstadt.de .