Retrieve-Refine-Calibrate: A Framework for Complex Claim Fact-Checking

TL;DR

This paper introduces the Retrieve-Refine-Calibrate framework leveraging large language models to improve complex claim fact-checking accuracy by reducing noise and calibrating verification confidence.

Contribution

It proposes a novel RRC framework that enhances evidence retrieval and verification calibration in fact-checking using LLMs, outperforming existing methods.

Findings

Achieves superior accuracy on HOVER and FEVEROUS-S datasets.

Effectively reduces irrelevant evidence in the verification process.

Improves confidence calibration in fact-checking predictions.

Abstract

Fact-checking aims to verify the truthfulness of a claim based on the retrieved evidence. Existing methods typically follow a decomposition paradigm, in which a claim is broken down into sub-claims that are individually verified. However, the decomposition paradigm may introduce noise to the verification process due to irrelevant entities or evidence, ultimately degrading verification accuracy. To address this problem, we propose a Retrieve-Refine-Calibrate (RRC) framework based on large language models (LLMs). Specifically, the framework first identifies the entities mentioned in the claim and retrieves evidence relevant to them. Then, it refines the retrieved evidence based on the claim to reduce irrelevant information. Finally, it calibrates the verification process by re-evaluating low-confidence predictions. Experiments on two popular fact-checking datasets (HOVER and FEVEROUS-S) demonstrate that our framework achieves superior performance compared with competitive baselines.