CrowdChecked: Detecting Previously Fact-Checked Claims in Social Media

TL;DR

This paper introduces CrowdChecked, a system that detects social media claims previously fact-checked by leveraging crowd-sourced data, using a large-scale dataset and an end-to-end learning framework to improve fact-checking accuracy.

Contribution

It presents a novel approach to utilize crowd-sourced social media claims and a large dataset for training fact-checking models, addressing data scarcity issues.

Findings

Achieved a two-point improvement over the state of the art on CLEF'21 dataset.

Created a large-scale dataset of 330,000 tweets with fact-checking articles.

Developed an end-to-end framework using modified self-adaptive training for noisy data.

Abstract

While there has been substantial progress in developing systems to automate fact-checking, they still lack credibility in the eyes of the users. Thus, an interesting approach has emerged: to perform automatic fact-checking by verifying whether an input claim has been previously fact-checked by professional fact-checkers and to return back an article that explains their decision. This is a sensible approach as people trust manual fact-checking, and as many claims are repeated multiple times. Yet, a major issue when building such systems is the small number of known tweet--verifying article pairs available for training. Here, we aim to bridge this gap by making use of crowd fact-checking, i.e., mining claims in social media for which users have responded with a link to a fact-checking article. In particular, we mine a large-scale collection of 330,000 tweets paired with a corresponding fact-checking article. We further propose an end-to-end framework to learn from this noisy data based on modified self-adaptive training, in a distant supervision scenario. Our experiments on the CLEF'21 CheckThat! test set show improvements over the state of the art by two points absolute. Our code and datasets are available at https://github.com/mhardalov/crowdchecked-claims