Robust Claim Verification Through Fact Detection

TL;DR

This paper introduces FactDetect, a method that improves claim verification robustness by extracting and utilizing short factual statements from evidence, enhancing performance and explainability in scientific claim verification.

Contribution

The paper presents a novel fact detection approach using LLMs and multitasking to boost claim verification accuracy and explainability, especially in scientific datasets.

Findings

FactDetect improves supervised claim verification F1 score by 15%.

AugFactDetect enhances zero-shot claim verification performance with an average of 17.3% gain.

The method demonstrates statistically significant improvements over baselines.

Abstract

Claim verification can be a challenging task. In this paper, we present a method to enhance the robustness and reasoning capabilities of automated claim verification through the extraction of short facts from evidence. Our novel approach, FactDetect, leverages Large Language Models (LLMs) to generate concise factual statements from evidence and label these facts based on their semantic relevance to the claim and evidence. The generated facts are then combined with the claim and evidence. To train a lightweight supervised model, we incorporate a fact-detection task into the claim verification process as a multitasking approach to improve both performance and explainability. We also show that augmenting FactDetect in the claim verification prompt enhances performance in zero-shot claim verification using LLMs. Our method demonstrates competitive results in the supervised claim verification model by 15% on the F1 score when evaluated for challenging scientific claim verification datasets. We also demonstrate that FactDetect can be augmented with claim and evidence for zero-shot prompting (AugFactDetect) in LLMs for verdict prediction. We show that AugFactDetect outperforms the baseline with statistical significance on three challenging scientific claim verification datasets with an average of 17.3% performance gain compared to the best performing baselines.