X-FACT: A New Benchmark Dataset for Multilingual Fact Checking

TL;DR

X-FACT is a comprehensive multilingual dataset for fact-checking in 25 languages, enabling evaluation of models' generalization and zero-shot capabilities, and highlighting the challenges in automated multilingual fact verification.

Contribution

We introduce X-FACT, the largest multilingual fact-checking dataset with an evaluation benchmark and develop models that incorporate metadata and evidence for improved verification.

Findings

Best model achieves around 40% F-score.

Dataset is challenging for current models.

Supports evaluation of out-of-domain and zero-shot performance.

Abstract

In this work, we introduce X-FACT: the largest publicly available multilingual dataset for factual verification of naturally existing real-world claims. The dataset contains short statements in 25 languages and is labeled for veracity by expert fact-checkers. The dataset includes a multilingual evaluation benchmark that measures both out-of-domain generalization, and zero-shot capabilities of the multilingual models. Using state-of-the-art multilingual transformer-based models, we develop several automated fact-checking models that, along with textual claims, make use of additional metadata and evidence from news stories retrieved using a search engine. Empirically, our best model attains an F-score of around 40%, suggesting that our dataset is a challenging benchmark for evaluation of multilingual fact-checking models.