FactIR: A Real-World Zero-shot Open-Domain Retrieval Benchmark for Fact-Checking

TL;DR

FactIR introduces a real-world, zero-shot open-domain retrieval benchmark for fact-checking, enabling better evaluation of retrieval models in complex, real-world scenarios.

Contribution

This paper presents FactIR, a new benchmark derived from real-world logs with human annotations, addressing the lack of comprehensive datasets for fact-checking retrieval tasks.

Findings

State-of-the-art models show limited performance in zero-shot fact-checking retrieval.

FactIR reveals the challenges of indirect reasoning in retrieval tasks.

Benchmark facilitates development of more effective fact-checking retrieval systems.

Abstract

The field of automated fact-checking increasingly depends on retrieving web-based evidence to determine the veracity of claims in real-world scenarios. A significant challenge in this process is not only retrieving relevant information, but also identifying evidence that can both support and refute complex claims. Traditional retrieval methods may return documents that directly address claims or lean toward supporting them, but often struggle with more complex claims requiring indirect reasoning. While some existing benchmarks and methods target retrieval for fact-checking, a comprehensive real-world open-domain benchmark has been lacking. In this paper, we present a real-world retrieval benchmark FactIR, derived from Factiverse production logs, enhanced with human annotations. We rigorously evaluate state-of-the-art retrieval models in a zero-shot setup on FactIR and offer insights for developing practical retrieval systems for fact-checking. Code and data are available at https://github.com/factiverse/factIR.