Are LLM-Powered Social Media Bots Realistic?

TL;DR

This paper explores the potential and realism of using Large Language Models to create social media bot networks, analyzing their properties and differences from real bots and humans to inform detection strategies.

Contribution

It introduces a method to generate synthetic LLM-powered social media bots and compares their network and linguistic features with real data.

Findings

Generated bot networks differ from real bots and humans in network structure.

Linguistic properties of LLM bots are distinguishable from natural human language.

Implications for bot detection and social media security.

Abstract

As Large Language Models (LLMs) become more sophisticated, there is a possibility to harness LLMs to power social media bots. This work investigates the realism of generating LLM-Powered social media bot networks. Through a combination of manual effort, network science and LLMs, we create synthetic bot agent personas, their tweets and their interactions, thereby simulating social media networks. We compare the generated networks against empirical bot/human data, observing that both network and linguistic properties of LLM-Powered Bots differ from Wild Bots/Humans. This has implications towards the detection and effectiveness of LLM-Powered Bots.