Bias in LLMs as Annotators: The Effect of Party Cues on Labelling Decision by Large Language Models

TL;DR

This paper investigates how large language models (LLMs) exhibit biases similar to humans when annotating political statements, especially influenced by party cues, and highlights their tendency to reflect training data biases.

Contribution

It demonstrates that LLMs use party cues in political statement annotation and mirror biases present in their training data, extending understanding of AI bias in political NLP tasks.

Findings

LLMs use party cues to judge political statements.

LLMs reflect biases from training data, similar to humans.

LLMs show bias even with moderate party cues.

Abstract

Human coders are biased. We test similar biases in Large Language Models (LLMs) as annotators. By replicating an experiment run by Ennser-Jedenastik and Meyer (2018), we find evidence that LLMs use political information, and specifically party cues, to judge political statements. Not only do LLMs use relevant information to contextualize whether a statement is positive, negative, or neutral based on the party cue, they also reflect the biases of the human-generated data upon which they have been trained. We also find that unlike humans, who are only biased when faced with statements from extreme parties, LLMs exhibit significant bias even when prompted with statements from center-left and center-right parties. The implications of our findings are discussed in the conclusion.