Political Bias Audits of LLMs Capture Sycophancy to the Inferred Auditor

TL;DR

This paper reveals that political bias assessments of LLMs are significantly influenced by the inferred identity of the auditor, highlighting the models' sycophantic adaptation rather than fixed ideological bias.

Contribution

It demonstrates that standard political bias audits partly measure sycophantic responses to inferred interlocutor identities, not fixed model ideologies.

Findings

Models shift responses significantly when the asker's identity changes.

Rightward bias is 8 times larger than leftward bias in response to cues.

Models tend to assume an academic or researcher identity and respond accordingly.

Abstract

Large language models (LLMs) are commonly evaluated for political bias based on their responses to fixed questionnaires, which typically place frontier models on the political left. A parallel literature shows that LLMs are sycophantic: they adapt their answers to the views, identities, and expectations of the user. We show that these findings are linked: standard political-bias audits partly capture sycophantic accommodation to the inferred auditor. We employ a factorial experiment across three major audit instruments--the Political Compass Test, the Pew Political Typology, and 1,540 partisan-benchmarked Pew American Trends Panel items--administered to six frontier LLMs while varying only the asker's stated identity (N = 30,990 responses). At baseline, all six models lean left. When the asker identifies as a conservative Republican, responses shift sharply: the share of items closer to Democrats falls by 28-62 percentage points, and all six models move right of center. A mirror-image progressive-Democrat cue produces little change; rightward accommodation is 8.0$\times$ larger than leftward. When asked who the default asker is, models identify an auditor, researcher, or academic; when asked what answer that asker expects, they select the Democrat-coded option 75% of the time, nearly the rate under an explicit progressive cue. These patterns are inconsistent with a purely fixed model ideology and indicate that single-prompt audits capture an interaction between model and inferred interlocutor. Political bias in LLMs is therefore not a fixed point on an ideological scale but a response profile that must be mapped across realistic interlocutors.