XFEVER: Exploring Fact Verification across Languages

TL;DR

This paper presents XFEVER, a multilingual fact verification dataset derived from FEVER, enabling cross-lingual verification research, and demonstrates baseline models' effectiveness and limitations across languages.

Contribution

Introduces the XFEVER dataset for cross-lingual fact verification, along with baseline models for zero-shot and translate-train scenarios.

Findings

Multilingual models can perform fact verification across languages.

Performance varies significantly by language.

Model calibration can be improved by analyzing prediction similarities.

Abstract

This paper introduces the Cross-lingual Fact Extraction and VERification (XFEVER) dataset designed for benchmarking the fact verification models across different languages. We constructed it by translating the claim and evidence texts of the Fact Extraction and VERification (FEVER) dataset into six languages. The training and development sets were translated using machine translation, whereas the test set includes texts translated by professional translators and machine-translated texts. Using the XFEVER dataset, two cross-lingual fact verification scenarios, zero-shot learning and translate-train learning, are defined, and baseline models for each scenario are also proposed in this paper. Experimental results show that the multilingual language model can be used to build fact verification models in different languages efficiently. However, the performance varies by language and is somewhat inferior to the English case. We also found that we can effectively mitigate model miscalibration by considering the prediction similarity between the English and target languages. The XFEVER dataset, code, and model checkpoints are available at https://github.com/nii-yamagishilab/xfever.