Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in LLMs

TL;DR

This study evaluates whether large language models reliably and consistently reflect political worldviews, revealing that larger models tend to align more with left-leaning perspectives but show variability across different policy issues.

Contribution

The paper introduces a comprehensive testing framework to assess the reliability and consistency of LLMs' political stances across multiple policy domains.

Findings

Reliability of political leanings increases with model size.

Larger models favor left-leaning views on environment and social welfare.

Models show inconsistent preferences in foreign policy and migration.

Abstract

Due to the widespread use of large language models (LLMs), we need to understand whether they embed a specific "worldview" and what these views reflect. Recent studies report that, prompted with political questionnaires, LLMs show left-liberal leanings (Feng et al., 2023; Motoki et al., 2024). However, it is as yet unclear whether these leanings are reliable (robust to prompt variations) and whether the leaning is consistent across policies and political leaning. We propose a series of tests which assess the reliability and consistency of LLMs' stances on political statements based on a dataset of voting-advice questionnaires collected from seven EU countries and annotated for policy issues. We study LLMs ranging in size from 7B to 70B parameters and find that their reliability increases with parameter count. Larger models show overall stronger alignment with left-leaning parties but differ among policy programs: They show a (left-wing) positive stance towards environment protection, social welfare state and liberal society but also (right-wing) law and order, with no consistent preferences in the areas of foreign policy and migration.