Perceived Political Bias in LLMs Reduces Persuasive Abilities

TL;DR

This study investigates how perceived political bias in large language models affects their ability to persuade users, finding that accusations of bias significantly reduce their persuasive effectiveness in politically charged contexts.

Contribution

The paper provides empirical evidence that perceived partisan bias diminishes LLMs' persuasive power, highlighting the importance of neutrality for effective AI communication.

Findings

Perceived bias reduces persuasion by 28%.

Warnings about bias increase user pushback.

Perceptions of partisanship constrain AI's persuasive impact.

Abstract

Conversational AI has been proposed as a scalable way to correct public misconceptions and spread misinformation. Yet its effectiveness may depend on perceptions of its political neutrality. As LLMs enter partisan conflict, elites increasingly portray them as ideologically aligned. We test whether these credibility attacks reduce LLM-based persuasion. In a preregistered U.S. survey experiment (N=2144), participants completed a three-round conversation with ChatGPT about a personally held economic policy misconception. Compared to a neutral control, a short message indicating that the LLM was biased against the respondent's party attenuated persuasion by 28%. Transcript analysis indicates that the warnings alter the interaction: respondents push back more and engage less receptively. These findings suggest that the persuasive impact of conversational AI is politically contingent, constrained by perceptions of partisan alignment.