A Detailed Factor Analysis for the Political Compass Test: Navigating Ideologies of Large Language Models

TL;DR

This paper investigates how prompt phrasing and fine-tuning influence political bias scores in large language models, revealing that these factors significantly affect results and questioning the validity of current bias measurement methods.

Contribution

It provides a detailed statistical analysis of factors affecting political bias assessments in LLMs, highlighting the impact of prompt phrasing and fine-tuning on test outcomes.

Findings

Prompt phrasing and fine-tuning significantly influence PCT scores.

Changes to standard generation parameters have minimal effect.

Models do not change responses to neutral prompts or texts.

Abstract

The Political Compass Test (PCT) and similar surveys are commonly used to assess political bias in auto-regressive LLMs. Our rigorous statistical experiments show that while changes to standard generation parameters have minimal effect on PCT scores, prompt phrasing and fine-tuning individually and together can significantly influence results. Interestingly, fine-tuning on politically rich vs. neutral datasets does not lead to different shifts in scores. We also generalize these findings to a similar popular test called 8 Values. Humans do not change their responses to questions when prompted differently (``answer this question'' vs ``state your opinion''), or after exposure to politically neutral text, such as mathematical formulae. But the fact that the models do so raises concerns about the validity of these tests for measuring model bias, and paves the way for deeper exploration into how political and social views are encoded in LLMs.