Characterizing AI-Generated Misinformation on Social Media

TL;DR

This study provides a large-scale empirical analysis of AI-generated misinformation on social media, revealing its unique characteristics, prevalence, and impact compared to traditional misinformation, with implications for platform moderation and future research.

Contribution

It is the first large-scale empirical study analyzing AI-generated misinformation on social media, highlighting its distinct features and virality compared to non-AI misinformation.

Findings

AI-generated misinformation is more entertainment-focused and positively toned.

It tends to originate from smaller user accounts.

AI-generated misinformation is more likely to go viral.

Abstract

AI-generated misinformation (e.g., deepfakes) poses a growing threat to information integrity on social media. However, prior research has largely focused on its potential societal consequences rather than its real-world prevalence. In this study, we conduct a large-scale empirical analysis of AI-generated misinformation on the social media platform X. Specifically, we analyze a dataset comprising N=91,452 misleading posts, both AI-generated and non-AI-generated, that have been identified and flagged through X's Community Notes platform. Our analysis yields four main findings: (i) AI-generated misinformation is more often centered on entertaining content and tends to exhibit a more positive sentiment than conventional forms of misinformation, (ii) it is more likely to originate from smaller user accounts, (iii) despite this, it is significantly more likely to go viral, and (iv) it is slightly less believable and harmful compared to conventional misinformation. Altogether, our findings highlight the unique characteristics of AI-generated misinformation on social media. We discuss important implications for platforms and future research.