How Large Language Models play humans in online conversations: a simulated study of the 2016 US politics on Reddit

TL;DR

This study evaluates GPT-4's ability to generate realistic Reddit comments during the 2016 US election, revealing its potential to influence political discourse and the challenges in distinguishing AI from human contributions.

Contribution

It demonstrates GPT-4's capacity to produce convincing political comments and analyzes their impact on online discussions, highlighting implications for discourse manipulation.

Findings

GPT-4 generates realistic, partisan comments

Artificial comments are semantically distinct but visually similar to real ones

AI can influence political narratives in online platforms

Abstract

Large Language Models (LLMs) have recently emerged as powerful tools for natural language generation, with applications spanning from content creation to social simulations. Their ability to mimic human interactions raises both opportunities and concerns, particularly in the context of politically relevant online discussions. In this study, we evaluate the performance of LLMs in replicating user-generated content within a real-world, divisive scenario: Reddit conversations during the 2016 US Presidential election. In particular, we conduct three different experiments, asking GPT-4 to generate comments by impersonating either real or artificial partisan users. We analyze the generated comments in terms of political alignment, sentiment, and linguistic features, comparing them against real user contributions and benchmarking against a null model. We find that GPT-4 is able to produce realistic comments, both in favor of or against the candidate supported by the community, yet tending to create consensus more easily than dissent. In addition we show that real and artificial comments are well separated in a semantically embedded space, although they are indistinguishable by manual inspection. Our findings provide insights on the potential use of LLMs to sneak into online discussions, influence political debate and shape political narratives, bearing broader implications of AI-driven discourse manipulation.