The Case for Claim Difficulty Assessment in Automatic Fact Checking

TL;DR

This paper highlights the importance of assessing claim difficulty in automated fact-checking, proposing a new prediction task to improve fact-checking accuracy and efficiency.

Contribution

It introduces the novel task of claim difficulty prediction and analyzes factors influencing claim difficulty to enhance fact-checking systems and practices.

Findings

Identified distinct types of claim difficulty.

Motivated the claim difficulty prediction task.

Discussed implications for dataset design and practical fact-checking.

Abstract

Fact-checking is the process of evaluating the veracity of claims (i.e., purported facts). In this opinion piece, we raise an issue that has received little attention in prior work -- that some claims are far more difficult to fact-check than others. We discuss the implications this has for both practical fact-checking and research on automated fact-checking, including task formulation and dataset design. We report a manual analysis undertaken to explore factors underlying varying claim difficulty and identify several distinct types of difficulty. We motivate this new claim difficulty prediction task as beneficial to both automated fact-checking and practical fact-checking organizations.