Navigating the Noisy Crowd: Finding Key Information for Claim Verification

TL;DR

This paper introduces EACon, a framework that improves claim verification by extracting key information from noisy evidence and verifying subclaims, significantly enhancing large language models' accuracy.

Contribution

EACon is a novel framework that uses keyword-based evidence abstraction and claim deconstruction to improve claim verification with noisy evidence.

Findings

EACon outperforms baseline methods on two datasets.

It significantly improves LLM accuracy in claim verification.

The framework effectively filters relevant information from noisy evidence.

Abstract

Claim verification is a task that involves assessing the truthfulness of a given claim based on multiple evidence pieces. Using large language models (LLMs) for claim verification is a promising way. However, simply feeding all the evidence pieces to an LLM and asking if the claim is factual does not yield good results. The challenge lies in the noisy nature of both the evidence and the claim: evidence passages typically contain irrelevant information, with the key facts hidden within the context, while claims often convey multiple aspects simultaneously. To navigate this "noisy crowd" of information, we propose EACon (Evidence Abstraction and Claim Deconstruction), a framework designed to find key information within evidence and verify each aspect of a claim separately. EACon first finds keywords from the claim and employs fuzzy matching to select relevant keywords for each raw evidence piece. These keywords serve as a guide to extract and summarize critical information into abstracted evidence. Subsequently, EACon deconstructs the original claim into subclaims, which are then verified against both abstracted and raw evidence individually. We evaluate EACon using two open-source LLMs on two challenging datasets. Results demonstrate that EACon consistently and substantially improve LLMs' performance in claim verification.