Decide less, communicate more: On the construct validity of end-to-end fact-checking in medicine

TL;DR

This paper examines the challenges of applying end-to-end fact-checking systems in medicine, highlighting fundamental issues and proposing a shift towards viewing fact-checking as an interactive communication process.

Contribution

It provides the first study of how clinical experts verify social media claims with medical evidence and identifies key challenges in automated medical fact-checking.

Findings

Difficulties in linking social media claims to scientific evidence

Ambiguities in underspecified claims and mismatched intentions

Subjectivity in veracity labels for medical claims

Abstract

Technological progress has led to concrete advancements in tasks that were regarded as challenging, such as automatic fact-checking. Interest in adopting these systems for public health and medicine has grown due to the high-stakes nature of medical decisions and challenges in critically appraising a vast and diverse medical literature. Evidence-based medicine connects to every individual, and yet the nature of it is highly technical, rendering the medical literacy of majority users inadequate to sufficiently navigate the domain. Such problems with medical communication ripen the ground for end-to-end fact-checking agents: check a claim against current medical literature and return with an evidence-backed verdict. And yet, such systems remain largely unused. In this position paper, developed with expert input, we present the first study examining how clinical experts verify real claims from social media by synthesizing medical evidence. In searching for this upper-bound, we reveal fundamental challenges in end-to-end fact-checking when applied to medicine: Difficulties connecting claims in the wild to scientific evidence in the form of clinical trials; ambiguities in underspecified claims mixed with mismatched intentions; and inherently subjective veracity labels. We argue that fact-checking should be approached as an interactive communication problem, rather than an end-to-end process.