Can social media provide early warning of retraction? Evidence from critical tweets identified by human annotation and large language models

TL;DR

This study investigates whether social media commentary, especially critical tweets, can serve as early warning signals for retracted scientific articles, highlighting the potential and limitations of AI-assisted detection methods.

Contribution

It demonstrates that critical tweets can precede retractions and evaluates the effectiveness of human annotation versus large language models in identifying problematic research discussions.

Findings

8.3% of retracted articles had critical tweets before retraction

Critical tweets are less common for non-retracted articles (1.5%)

AI models partially align with human annotations, indicating cautious use is needed

Abstract

Timely detection of problematic research is essential for safeguarding scientific integrity. To explore whether social media commentary can serve as an early indicator of potentially problematic articles, this study analysed 3,815 tweets referencing 604 retracted articles and 3,373 tweets referencing 668 comparable non-retracted articles. Tweets critical of the articles were identified through both human annotation and large language models (LLMs). Human annotation revealed that 8.3% of retracted articles were associated with at least one critical tweet prior to retraction, compared to only 1.5% of non-retracted articles, highlighting the potential of tweets as early warning signals of retraction. However, critical tweets identified by LLMs (GPT-4o mini, Gemini 2.0 Flash-Lite, and Claude 3.5 Haiku) only partially aligned with human annotation, suggesting that fully automated monitoring of post-publication discourse should be applied with caution. A human-AI collaborative approach may offer a more reliable and scalable alternative, with human expertise helping to filter out tweets critical of issues unrelated to the research integrity of the articles. Overall, this study provides insights into how social media signals, combined with generative AI technologies, may support efforts to strengthen research integrity.