Claim Extraction for Fact-Checking: Data, Models, and Automated Metrics

TL;DR

This paper investigates claim extraction for fact-checking using various text generation models, introduces a new dataset and evaluation framework, and validates automated metrics against human judgment.

Contribution

It introduces the FEVERFact dataset, compares multiple models for claim extraction, and develops an evaluation framework aligned with human grading.

Findings

Models' rankings consistent with human judgments on $F_{fact}$

Evaluation metrics closely approximate human grading

Proposed framework effectively assesses claim extraction quality

Abstract

In this paper, we explore the problem of Claim Extraction using one-to-many text generation methods, comparing LLMs, small summarization models finetuned for the task, and a previous NER-centric baseline QACG. As the current publications on Claim Extraction, Fact Extraction, Claim Generation and Check-worthy Claim Detection are quite scattered in their means and terminology, we compile their common objectives, releasing the FEVERFact dataset, with 17K atomic factual claims extracted from 4K contextualised Wikipedia sentences, adapted from the original FEVER. We compile the known objectives into an Evaluation framework of: Atomicity, Fluency, Decontextualization, Faithfulness checked for each generated claim separately, and Focus and Coverage measured against the full set of predicted claims for a single input. For each metric, we implement a scale using a reduction to an already-explored NLP task. We validate our metrics against human grading of generic claims, to see that the model ranking on $F_{fact}$, our hardest metric, did not change and the evaluation framework approximates human grading very closely in terms of $F_1$ and RMSE.