Crowdsourced Fact-Checking at Twitter: How Does the Crowd Compare With Experts?

TL;DR

This study evaluates Twitter's Birdwatch crowdsourced fact-checking program, comparing its effectiveness and efficiency to expert fact-checkers, revealing strengths in scalability but inconsistencies in accuracy and content selection.

Contribution

First large-scale empirical analysis of a real-world crowdsourced fact-checking system deployed by Twitter, highlighting its potential and limitations compared to expert verification.

Findings

Crowdsourcing can be effective and scalable for fact-checking in some settings.

Crowd and experts differ in content selection and resource retrieval.

Crowdsourcing shows advantages in scalability and efficiency over experts.

Abstract

Fact-checking is one of the effective solutions in fighting online misinformation. However, traditional fact-checking is a process requiring scarce expert human resources, and thus does not scale well on social media because of the continuous flow of new content to be checked. Methods based on crowdsourcing have been proposed to tackle this challenge, as they can scale with a smaller cost, but, while they have shown to be feasible, have always been studied in controlled environments. In this work, we study the first large-scale effort of crowdsourced fact-checking deployed in practice, started by Twitter with the Birdwatch program. Our analysis shows that crowdsourcing may be an effective fact-checking strategy in some settings, even comparable to results obtained by human experts, but does not lead to consistent, actionable results in others. We processed 11.9k tweets verified by the Birdwatch program and report empirical evidence of i) differences in how the crowd and experts select content to be fact-checked, ii) how the crowd and the experts retrieve different resources to fact-check, and iii) the edge the crowd shows in fact-checking scalability and efficiency as compared to expert checkers.