FIRE: Fact-checking with Iterative Retrieval and Verification

TL;DR

FIRE is an innovative fact-checking framework that iteratively retrieves evidence and verifies claims, significantly reducing costs while maintaining or improving accuracy in large-scale fact-checking tasks.

Contribution

FIRE introduces an agent-based, iterative approach to fact-checking that unifies evidence retrieval and verification, improving efficiency and cost-effectiveness over traditional methods.

Findings

Achieves slightly better accuracy than existing frameworks.

Reduces language model costs by 7.6 times on average.

Lowers search costs by 16.5 times.

Abstract

Fact-checking long-form text is challenging, and it is therefore common practice to break it down into multiple atomic claims. The typical approach to fact-checking these atomic claims involves retrieving a fixed number of pieces of evidence, followed by a verification step. However, this method is usually not cost-effective, as it underutilizes the verification model's internal knowledge of the claim and fails to replicate the iterative reasoning process in human search strategies. To address these limitations, we propose FIRE, a novel agent-based framework that integrates evidence retrieval and claim verification in an iterative manner. Specifically, FIRE employs a unified mechanism to decide whether to provide a final answer or generate a subsequent search query, based on its confidence in the current judgment. We compare FIRE with other strong fact-checking frameworks and find that it achieves slightly better performance while reducing large language model (LLM) costs by an average of 7.6 times and search costs by 16.5 times. These results indicate that FIRE holds promise for application in large-scale fact-checking operations. Our code is available at https://github.com/mbzuai-nlp/fire.git.