Consensus Stability of Community Notes on X

TL;DR

This study analyzes how community notes on X change in helpfulness ratings after being displayed, revealing that polarization among raters can lead to note disappearance and affect system stability.

Contribution

It provides the first large-scale analysis of post-display rating dynamics and polarization effects in community-based fact-checking systems.

Findings

30.2% of helpful notes lose helpfulness and disappear.

Display triggers increased rating volume and polarization.

Dissimilar raters contribute significantly to note disappearance.

Abstract

Community-based fact-checking systems, such as Community Notes on X (formerly Twitter), aim to mitigate online misinformation by surfacing annotations judged helpful by contributors with diverse viewpoints. While prior work has shown that the platform's bridging-based algorithm effectively selects helpful notes at the time of display, little is known about how evaluations change after notes become visible. Using a large-scale dataset of 437,396 community notes and 35 million ratings from over 580,000 contributors, we examine the stability of helpful notes and the rating dynamics that follow their initial display. We find that 30.2% of displayed notes later lose their helpful status and disappear. Using interrupted time series models, we further show that note display triggers a sharp increase in rating volume and a significant shift in rating leaning, but these effects differ across rater groups. Contributors with viewpoints similar to note authors tend to increase supportive ratings, while dissimilar contributors increase negative ratings, producing systematic post-display polarization. Counterfactual analyses suggest that this post-display polarization, particularly from dissimilar raters, plays a substantial role in note disappearance. These findings highlight the vulnerability of consensus-based fact-checking systems to polarized rating behavior and suggest pathways for improving their resilience.