Retrieval Augmented Fact Verification by Synthesizing Contrastive Arguments

TL;DR

RAFTS enhances fact verification by retrieving relevant evidence and synthesizing contrasting arguments, leading to improved accuracy with smaller language models without relying on black-box APIs.

Contribution

The paper introduces RAFTS, a retrieval-augmented framework that synthesizes supporting and refuting arguments from retrieved evidence for more accurate fact verification.

Findings

RAFTS outperforms GPT-based methods with a 7B LLM.

Effective retrieval and contrastive argument synthesis improve verification accuracy.

Significant gains over supervised and baseline methods without complex prompts.

Abstract

The rapid propagation of misinformation poses substantial risks to public interest. To combat misinformation, large language models (LLMs) are adapted to automatically verify claim credibility. Nevertheless, existing methods heavily rely on the embedded knowledge within LLMs and / or black-box APIs for evidence collection, leading to subpar performance with smaller LLMs or upon unreliable context. In this paper, we propose retrieval augmented fact verification through the synthesis of contrasting arguments (RAFTS). Upon input claims, RAFTS starts with evidence retrieval, where we design a retrieval pipeline to collect and re-rank relevant documents from verifiable sources. Then, RAFTS forms contrastive arguments (i.e., supporting or refuting) conditioned on the retrieved evidence. In addition, RAFTS leverages an embedding model to identify informative demonstrations, followed by in-context prompting to generate the prediction and explanation. Our method effectively retrieves relevant documents as evidence and evaluates arguments from varying perspectives, incorporating nuanced information for fine-grained decision-making. Combined with informative in-context examples as prior, RAFTS achieves significant improvements to supervised and LLM baselines without complex prompts. We demonstrate the effectiveness of our method through extensive experiments, where RAFTS can outperform GPT-based methods with a significantly smaller 7B LLM.