GermanPartiesQA: Benchmarking Commercial Large Language Models and AI Companions for Political Alignment and Sycophancy

TL;DR

This paper introduces GermanPartiesQA, a benchmark for evaluating commercial large language models' political alignment and bias, revealing their factual limitations, ideological tendencies, and steerability in role-playing scenarios.

Contribution

It presents a new benchmark dataset and evaluation methodology for assessing political alignment and biases in commercial LLMs used in decision support tools.

Findings

LLMs have limited accuracy in representing factual party positions.

Models exhibit consistent ideological alignment patterns.

Models' responses reflect persona-based steerability, not true sycophancy.

Abstract

Large language models (LLMs) are increasingly shaping citizens' information ecosystems. Products incorporating LLMs, such as chatbots and AI Companions, are now widely used for decision support and information retrieval, including in sensitive domains, raising concerns about hidden biases and growing potential to shape individual decisions and public opinion. This paper introduces GermanPartiesQA, a benchmark of 418 political statements from German Voting Advice Applications across 11 elections to evaluate six commercial LLMs. We evaluate their political alignment based on role-playing experiments with political personas. Our evaluation reveals three specific findings: (1) Factual limitations: LLMs show limited ability to accurately generate factual party positions, particularly for centrist parties. (2) Model-specific ideological alignment: We identify consistent alignment patterns and the degree of political steerability for each model across temperature settings and experiments. (3) Claim of sycophancy: While models adjust to political personas during role-play, we find this reflects persona-based steerability rather than the increasingly popular, yet contested concept of sycophancy. Our study contributes to evaluating the political alignment of closed-source LLMs that are increasingly embedded in electoral decision support tools and AI Companion chatbots.