The Promises and Pitfalls of LLM Annotations in Dataset Labeling: a Case Study on Media Bias Detection

TL;DR

This paper explores using Large Language Models to automate media bias annotation, creating a large dataset that enables training effective bias classifiers while analyzing the benefits and limitations of this approach.

Contribution

It introduces annolexical, the first large-scale media bias dataset annotated by LLMs, and demonstrates that classifiers trained on this data outperform LLM annotators and rival human-labeled data.

Findings

Classifier trained on LLM-annotated data outperforms LLM annotators by 5-9% MCC.

The dataset enables cost-effective media bias classification.

Limitations and trade-offs of LLM-based annotation are identified.

Abstract

High annotation costs from hiring or crowdsourcing complicate the creation of large, high-quality datasets needed for training reliable text classifiers. Recent research suggests using Large Language Models (LLMs) to automate the annotation process, reducing these costs while maintaining data quality. LLMs have shown promising results in annotating downstream tasks like hate speech detection and political framing. Building on the success in these areas, this study investigates whether LLMs are viable for annotating the complex task of media bias detection and whether a downstream media bias classifier can be trained on such data. We create annolexical, the first large-scale dataset for media bias classification with over 48000 synthetically annotated examples. Our classifier, fine-tuned on this dataset, surpasses all of the annotator LLMs by 5-9 percent in Matthews Correlation Coefficient (MCC) and performs close to or outperforms the model trained on human-labeled data when evaluated on two media bias benchmark datasets (BABE and BASIL). This study demonstrates how our approach significantly reduces the cost of dataset creation in the media bias domain and, by extension, the development of classifiers, while our subsequent behavioral stress-testing reveals some of its current limitations and trade-offs.