Measuring Political Stance and Consistency in Large Language Models

TL;DR

This paper evaluates how large language models reflect political biases and stance consistency across various issues, revealing variability in their responses and highlighting potential biases influenced by language and prompting methods.

Contribution

It introduces a systematic assessment of political stance and consistency in multiple LLMs using diverse prompting techniques, uncovering biases and stability patterns.

Findings

Models often adopt opposing stances on issues.

Prompting can influence model positions, but some stances remain stable.

Language used in prompts affects model support for certain issues.

Abstract

With the incredible advancements in Large Language Models (LLMs), many people have started using them to satisfy their information needs. However, utilizing LLMs might be problematic for political issues where disagreement is common and model outputs may reflect training-data biases or deliberate alignment choices. To better characterize such behavior, we assess the stances of nine LLMs on 24 politically sensitive issues using five prompting techniques. We find that models often adopt opposing stances on several issues; some positions are malleable under prompting, while others remain stable. Among the models examined, Grok-3-mini is the most persistent, whereas Mistral-7B is the least. For issues involving countries with different languages, models tend to support the side whose language is used in the prompt. Notably, no prompting technique alters model stances on the Qatar blockade or the oppression of Palestinians. We hope these findings raise user awareness when seeking political guidance from LLMs and encourage developers to address these concerns.