Human-in-the-loop Evaluation for Early Misinformation Detection: A Case Study of COVID-19 Treatments

TL;DR

This paper introduces a human-in-the-loop framework for early detection of COVID-19 misinformation on social media, combining NLP techniques with human review to improve fact-checking accuracy.

Contribution

It proposes a novel evaluation framework and baseline system for identifying and verifying COVID-19 treatment misinformation using NLP and human review.

Findings

Developed a check-worthy claim extraction and ranking method

Created stance classifiers to identify supporting tweets

Made data and guidelines publicly available for system evaluation

Abstract

We present a human-in-the-loop evaluation framework for fact-checking novel misinformation claims and identifying social media messages that support them. Our approach extracts check-worthy claims, which are aggregated and ranked for review. Stance classifiers are then used to identify tweets supporting novel misinformation claims, which are further reviewed to determine whether they violate relevant policies. To demonstrate the feasibility of our approach, we develop a baseline system based on modern NLP methods for human-in-the-loop fact-checking in the domain of COVID-19 treatments. We make our data and detailed annotation guidelines available to support the evaluation of human-in-the-loop systems that identify novel misinformation directly from raw user-generated content.