Investigating Differences in Crowdsourced News Credibility Assessment: Raters, Tasks, and Expert Criteria

TL;DR

This study compares crowdsourced and expert news credibility assessments on climate-related articles, revealing differences influenced by crowd demographics, task scope, and expert criteria, and suggests tailored task designs to improve crowd accuracy.

Contribution

It provides a detailed analysis of how crowd and expert ratings differ and identifies factors affecting these differences, proposing tailored crowd task designs to align with expert criteria.

Findings

Crowd ratings vary by demographics and political leaning.

Task scope influences credibility assessment accuracy.

Expert criteria differ between journalism and science experts.

Abstract

Misinformation about critical issues such as climate change and vaccine safety is oftentimes amplified on online social and search platforms. The crowdsourcing of content credibility assessment by laypeople has been proposed as one strategy to combat misinformation by attempting to replicate the assessments of experts at scale. In this work, we investigate news credibility assessments by crowds versus experts to understand when and how ratings between them differ. We gather a dataset of over 4,000 credibility assessments taken from 2 crowd groups---journalism students and Upwork workers---as well as 2 expert groups---journalists and scientists---on a varied set of 50 news articles related to climate science, a topic with widespread disconnect between public opinion and expert consensus. Examining the ratings, we find differences in performance due to the makeup of the crowd, such as rater demographics and political leaning, as well as the scope of the tasks that the crowd is assigned to rate, such as the genre of the article and partisanship of the publication. Finally, we find differences between expert assessments due to differing expert criteria that journalism versus science experts use---differences that may contribute to crowd discrepancies, but that also suggest a way to reduce the gap by designing crowd tasks tailored to specific expert criteria. From these findings, we outline future research directions to better design crowd processes that are tailored to specific crowds and types of content.