Recon, Answer, Verify: Agents in Search of Truth

TL;DR

This paper introduces a realistic political claim dataset for fact checking, analyzes LLM performance drops in this setting, and proposes RAV, an agentic framework that improves verification accuracy across domains.

Contribution

It presents PFO, a more realistic benchmark dataset for fact checking, and introduces RAV, a novel agent-based system that enhances verification performance across multiple domains.

Findings

LLMs' performance drops by 22% on PFO compared to unfiltered data.

RAV outperforms state-of-the-art methods on multiple benchmarks.

RAV shows minimal performance drop of 16.3% in macro f1 when using filtered data.

Abstract

Automated fact checking with large language models (LLMs) offers a scalable alternative to manual verification. Evaluating fact checking is challenging as existing benchmark datasets often include post claim analysis and annotator cues, which are absent in real world scenarios where claims are fact checked immediately after being made. This limits the realism of current evaluations. We present Politi Fact Only (PFO), a 5 class benchmark dataset of 2,982 political claims from politifact.com, where all post claim analysis and annotator cues have been removed manually. This ensures that models are evaluated using only the information that would have been available prior to the claim's verification. Evaluating LLMs on PFO, we see an average performance drop of 22% in terms of macro f1 compared to PFO's unfiltered version. Based on the identified challenges of the existing LLM based fact checking system, we propose RAV (Recon Answer Verify), an agentic framework with three agents: question generator, answer generator, and label generator. Our pipeline iteratively generates and answers sub questions to verify different aspects of the claim before finally generating the label. RAV generalizes across domains and label granularities, and it outperforms state of the art approaches on well known baselines RAWFC (fact checking, 3 class) by 25.28%, and on HOVER (encyclopedia, 2 class) by 1.54% on 2 hop, 4.94% on 3 hop, and 1.78% on 4 hop, sub categories respectively. RAV shows the least performance drop compared to baselines of 16.3% in macro f1 when we compare PFO with its unfiltered version.