Analyzing Political Bias in LLMs via Target-Oriented Sentiment Classification

TL;DR

This paper introduces a novel entropy-based metric to analyze political bias in large language models by examining target-oriented sentiment prediction inconsistencies across multiple languages and model sizes.

Contribution

It proposes a new bias analysis method that leverages prediction variability with diverse politician names, revealing biases and language effects in LLMs.

Findings

Bias exists across all tested models and languages.

Larger models show stronger biases and consistency.

Western languages exhibit higher bias intensity.

Abstract

Political biases encoded by LLMs might have detrimental effects on downstream applications. Existing bias analysis methods rely on small-size intermediate tasks (questionnaire answering or political content generation) and rely on the LLMs themselves for analysis, thus propagating bias. We propose a new approach leveraging the observation that LLM sentiment predictions vary with the target entity in the same sentence. We define an entropy-based inconsistency metric to encode this prediction variability. We insert 1319 demographically and politically diverse politician names in 450 political sentences and predict target-oriented sentiment using seven models in six widely spoken languages. We observe inconsistencies in all tested combinations and aggregate them in a statistically robust analysis at different granularity levels. We observe positive and negative bias toward left and far-right politicians and positive correlations between politicians with similar alignment. Bias intensity is higher for Western languages than for others. Larger models exhibit stronger and more consistent biases and reduce discrepancies between similar languages. We partially mitigate LLM unreliability in target-oriented sentiment classification (TSC) by replacing politician names with fictional but plausible counterparts.