Benchmarking Political Persuasion Risks Across Frontier Large Language Models

TL;DR

This study benchmarks the persuasive risks of frontier large language models across bipartisan issues, revealing significant variability in their ability to influence political opinions and the impact of different prompting strategies.

Contribution

It introduces a comprehensive framework for assessing and comparing the political persuasion risks of state-of-the-art LLMs, highlighting model-dependent effects of prompts.

Findings

LLMs outperform standard campaign ads in persuasion.

Claude models are most persuasive, Grok least.

Prompt effectiveness varies by model, increasing persuasion for some and decreasing for others.

Abstract

Concerns persist regarding the capacity of Large Language Models (LLMs) to sway political views. Although prior research has claimed that LLMs are not more persuasive than standard political campaign practices, the recent rise of frontier models warrants further study. In two survey experiments (N=19,145) across bipartisan issues and stances, we evaluate seven state-of-the-art LLMs developed by Anthropic, OpenAI, Google, and xAI. We find that LLMs outperform standard campaign advertisements, with heterogeneity in performance across models. Specifically, Claude models exhibit the highest persuasiveness, while Grok exhibits the lowest. The results are robust across issues and stances. Moreover, in contrast to the findings in Hackenburg et al. (2025b) and Lin et al. (2025) that information-based prompts boost persuasiveness, we find that the effectiveness of information-based prompts is model-dependent: they increase the persuasiveness of Claude and Grok while substantially reducing that of GPT. We introduce a data-driven and strategy-agnostic LLM-assisted conversation analysis approach to identify and assess underlying persuasive strategies. Our work benchmarks the persuasive risks of frontier models and provides a framework for cross-model comparative risk assessment.