Examining the Influence of Political Bias on Large Language Model Performance in Stance Classification

TL;DR

This paper investigates how political biases in large language models affect their accuracy in stance classification tasks, revealing dataset-level differences and challenges with ambiguous targets.

Contribution

It systematically analyzes the impact of political biases on LLM performance in stance classification across multiple datasets, models, and prompts, highlighting bias-related performance variations.

Findings

Performance varies significantly across politically oriented datasets.

Models and prompts show similar performance across datasets.

Ambiguity in targets reduces stance classification accuracy.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in executing tasks based on natural language queries. However, these models, trained on curated datasets, inherently embody biases ranging from racial to national and gender biases. It remains uncertain whether these biases impact the performance of LLMs for certain tasks. In this study, we investigate the political biases of LLMs within the stance classification task, specifically examining whether these models exhibit a tendency to more accurately classify politically-charged stances. Utilizing three datasets, seven LLMs, and four distinct prompting schemes, we analyze the performance of LLMs on politically oriented statements and targets. Our findings reveal a statistically significant difference in the performance of LLMs across various politically oriented stance classification tasks. Furthermore, we observe that this difference primarily manifests at the dataset level, with models and prompting schemes showing statistically similar performances across different stance classification datasets. Lastly, we observe that when there is greater ambiguity in the target the statement is directed towards, LLMs have poorer stance classification accuracy. Code & Dataset: http://doi.org/10.5281/zenodo.12938478