Can the Crowd Judge Truthfulness? A Longitudinal Study on Recent Misinformation about COVID-19

TL;DR

This study evaluates the effectiveness of crowdsourcing in assessing COVID-19 misinformation, showing that non-experts can reliably judge truthfulness and that judgment quality varies over time and with worker background.

Contribution

It provides a longitudinal analysis of crowdsourcing for misinformation detection during a pandemic, highlighting factors affecting judgment accuracy and reliability.

Findings

Crowd workers can accurately assess COVID-19 statement truthfulness.

Aggregation methods improve overall judgment quality.

Time span influences judgment accuracy for both novice and experienced workers.

Abstract

Recently, the misinformation problem has been addressed with a crowdsourcing-based approach: to assess the truthfulness of a statement, instead of relying on a few experts, a crowd of non-expert is exploited. We study whether crowdsourcing is an effective and reliable method to assess truthfulness during a pandemic, targeting statements related to COVID-19, thus addressing (mis)information that is both related to a sensitive and personal issue and very recent as compared to when the judgment is done. In our experiments, crowd workers are asked to assess the truthfulness of statements, and to provide evidence for the assessments. Besides showing that the crowd is able to accurately judge the truthfulness of the statements, we report results on workers behavior, agreement among workers, effect of aggregation functions, of scales transformations, and of workers background and bias. We perform a longitudinal study by re-launching the task multiple times with both novice and experienced workers, deriving important insights on how the behavior and quality change over time. Our results show that: workers are able to detect and objectively categorize online (mis)information related to COVID-19; both crowdsourced and expert judgments can be transformed and aggregated to improve quality; worker background and other signals (e.g., source of information, behavior) impact the quality of the data. The longitudinal study demonstrates that the time-span has a major effect on the quality of the judgments, for both novice and experienced workers. Finally, we provide an extensive failure analysis of the statements misjudged by the crowd-workers.