The Laziness of the Crowd: Effort Aversion Among Raters Risks Undermining the Efficacy of X's Community Notes Program

TL;DR

This study reveals that crowdsourced moderation systems like Twitter's Community Notes are less effective at addressing plausible misinformation because raters tend to avoid engaging with claims that require more cognitive effort to evaluate, risking the system's overall efficacy.

Contribution

The paper identifies effort aversion among raters as a key factor undermining the effectiveness of crowdsourced fact-checking, and proposes design solutions to mitigate this issue.

Findings

Less helpful notes are more common for easier-to-fact-check claims.

Claims that are more plausible and harder to refute receive fewer helpful notes.

Effort aversion leads to systematic failure in addressing plausible misinformation.

Abstract

Crowdsourced moderation systems like Twitter/X's Community Notes program have been proposed as scalable alternatives to professional fact-checkers for combating online misinformation. While prior research has examined the effectiveness of such systems in reducing engagement with false content and their vulnerability to partisan bias, we identify a previously untested mechanism linking fact-check difficulty to systematic non-participation by crowdsourced raters. We hypothesize that claims requiring less cognitive effort to evaluate, specifically, those that are obviously false and easy to refute, are more likely to receive public notes than claims that are more plausible and require greater effort to debunk. Using eighteen months of vaccine-related Community Notes data (2,250 posts) and ratings from 382 survey participants, we show that claims perceived as more difficult to fact-check are significantly less likely to receive notes that achieve ``helpful''/public status. Following the conduct of additional analyses and a fact-checking process utilizing an LLM pipeline to help rule out alternative explanations, we interpret this pattern as consistent with an unwillingness among raters to invest the mental effort required to evaluate and rate notes for more plausible misinformation. These findings suggest that crowdsourced moderation may systematically fail to address the forms of plausible misinformation which are most likely to deceive. We discuss implications for platform design and propose mechanisms to mitigate this difficulty penalty in crowdsourced content moderation systems.