AI Fact-Checking in the Wild: A Field Evaluation of LLM-Written Community Notes on X

TL;DR

This study evaluates the real-world performance of an LLM-based fact-checking system deployed on X, showing it can produce helpful, cross-partisan notes at scale, but emphasizing the importance of platform-specific evaluation methods.

Contribution

First field evaluation of an LLM-driven fact-checking pipeline on a live social media platform, comparing its effectiveness to human notes using real platform data.

Findings

LLM notes received more positive ratings than human notes across diverse raters.

LLM notes achieved higher helpfulness scores among raters evaluating all notes on a post.

The study highlights the importance of platform-aware evaluation for deploying AI fact-checkers.

Abstract

Large language models show promising capabilities for contextual fact-checking on social media: they can verify contested claims through deep research, synthesize evidence from multiple sources, and draft explanations at scale. However, prior work evaluates LLM fact-checking only in controlled settings using benchmarks or crowdworker judgments, leaving open how these systems perform in authentic platform environments. We present the first field evaluation of LLM-based fact-checking deployed on a live social media platform, testing performance directly through X Community Notes' AI writer feature over a three-month period. Our LLM writer, a multi-step pipeline that handles multimodal content (text, images, and videos), conducts web and platform-native search, and writes contextual notes, was deployed to write 1,614 notes on 1,597 tweets and compared against 1,332 human-written notes on the same tweets using 108,169 ratings from 42,521 raters. Direct comparison of note-level platform outcomes is complicated by differences in submission timing and rating exposure between LLM and human notes; we therefore pursue two complementary strategies: a rating-level analysis modeling individual rater evaluations, and a note-level analysis that equalizes rater exposure across note types. Rating-level analysis shows that LLM notes receive more positive ratings than human notes across raters with different political viewpoints, suggesting the potential for LLM-written notes to achieve the cross-partisan consensus. Note-level analysis confirms this advantage: among raters who evaluated all notes on the same post, LLM notes achieve significantly higher helpfulness scores. Our findings demonstrate that LLMs can contribute high-quality, broadly helpful fact-checking at scale, while highlighting that real-world evaluation requires careful attention to platform dynamics absent from controlled settings.