WiCE: Real-World Entailment for Claims in Wikipedia

TL;DR

WiCE is a new dataset for fine-grained entailment in Wikipedia claims, enabling better fact verification and evidence retrieval, and demonstrating the limitations of current models in real-world claim verification tasks.

Contribution

The paper introduces WiCE, a novel dataset with sub-sentence entailment and evidence support, along with an automatic claim decomposition method using GPT-3.5 to enhance entailment model performance.

Findings

WiCE improves entailment model performance across multiple datasets.

Claim decomposition with GPT-3.5 enhances verification accuracy.

Existing models struggle with real-world claim verification and retrieval.

Abstract

Textual entailment models are increasingly applied in settings like fact-checking, presupposition verification in question answering, or summary evaluation. However, these represent a significant domain shift from existing entailment datasets, and models underperform as a result. We propose WiCE, a new fine-grained textual entailment dataset built on natural claim and evidence pairs extracted from Wikipedia. In addition to standard claim-level entailment, WiCE provides entailment judgments over sub-sentence units of the claim, and a minimal subset of evidence sentences that support each subclaim. To support this, we propose an automatic claim decomposition strategy using GPT-3.5 which we show is also effective at improving entailment models' performance on multiple datasets at test time. Finally, we show that real claims in our dataset involve challenging verification and retrieval problems that existing models fail to address.