A Benchmark for Open-Domain Numerical Fact-Checking Enhanced by Claim Decomposition

TL;DR

This paper introduces QuanTemp++, a benchmark dataset for open-domain numerical fact-checking that emphasizes realistic evidence retrieval through claim decomposition, aiming to improve automated verification of numerical claims.

Contribution

The paper presents QuanTemp++, a new dataset with natural numerical claims and evidence collected via claim decomposition, avoiding temporal leakage and improving realism in fact-checking benchmarks.

Findings

Claim decomposition improves evidence relevance.

Verification accuracy benefits from realistic retrieval settings.

Analysis of retrieval paradigms highlights their impact on verification outcomes.

Abstract

Fact-checking numerical claims is critical as the presence of numbers provide mirage of veracity despite being fake potentially causing catastrophic impacts on society. The prior works in automatic fact verification do not primarily focus on natural numerical claims. A typical human fact-checker first retrieves relevant evidence addressing the different numerical aspects of the claim and then reasons about them to predict the veracity of the claim. Hence, the search process of a human fact-checker is a crucial skill that forms the foundation of the verification process. Emulating a real-world setting is essential to aid in the development of automated methods that encompass such skills. However, existing benchmarks employ heuristic claim decomposition approaches augmented with weakly supervised web search to collect evidences for verifying claims. This sometimes results in less relevant evidences and noisy sources with temporal leakage rendering a less realistic retrieval setting for claim verification. Hence, we introduce QuanTemp++: a dataset consisting of natural numerical claims, an open domain corpus, with the corresponding relevant evidence for each claim. The evidences are collected through a claim decomposition process approximately emulating the approach of human fact-checker and veracity labels ensuring there is no temporal leakage. Given this dataset, we also characterize the retrieval performance of key claim decomposition paradigms. Finally, we observe their effect on the outcome of the verification pipeline and draw insights. The code for data pipeline along with link to data can be found at https://github.com/VenkteshV/QuanTemp_Plus