Who Checks the Checkers? Exploring Source Credibility in Twitter's Community Notes

TL;DR

This study analyzes Twitter's Community Notes feature, revealing biases in source credibility and how source factuality affects public agreement and helpfulness of fact-checks, highlighting strengths and biases in crowd-sourced misinformation mitigation.

Contribution

It provides a multi-faceted analysis of source credibility and audience perception in Twitter's Community Notes, uncovering biases and the influence of source factuality.

Findings

Majority of cited sources are left-leaning news outlets with high factuality.

Left-biased and low factuality sources tend to validate tweets more.

Source factuality significantly impacts public agreement and perceived helpfulness.

Abstract

In recent years, the proliferation of misinformation on social media platforms has become a significant concern. Initially designed for sharing information and fostering social connections, platforms like Twitter (now rebranded as X) have also unfortunately become conduits for spreading misinformation. To mitigate this, these platforms have implemented various mechanisms, including the recent suggestion to use crowd-sourced non-expert fact-checkers to enhance the scalability and efficiency of content vetting. An example of this is the introduction of Community Notes on Twitter. While previous research has extensively explored various aspects of Twitter tweets, such as information diffusion, sentiment analytics and opinion summarization, there has been a limited focus on the specific feature of Twitter Community Notes, despite its potential role in crowd-sourced fact-checking. Prior research on Twitter Community Notes has involved empirical analysis of the feature's dataset and comparative studies that also include other methods like expert fact-checking. Distinguishing itself from prior works, our study covers a multi-faceted analysis of sources and audience perception within Community Notes. We find that the majority of cited sources are news outlets that are left-leaning and are of high factuality, pointing to a potential bias in the platform's community fact-checking. Left biased and low factuality sources validate tweets more, while Center sources are used more often to refute tweet content. Additionally, source factuality significantly influences public agreement and helpfulness of the notes, highlighting the effectiveness of the Community Notes Ranking algorithm. These findings showcase the impact and biases inherent in community-based fact-checking initiatives.