Investigating Bias in LLM-Based Bias Detection: Disparities between LLMs and Human Perception

TL;DR

This paper examines biases within large language models used for media bias detection, analyzing their impact and proposing strategies to mitigate bias, thereby improving fairness and reliability in AI-driven bias assessment.

Contribution

It uniquely investigates internal biases of LLMs in bias detection tasks and introduces debiasing techniques like prompt engineering and fine-tuning.

Findings

LLMs exhibit significant biases in political and topical predictions.

Bias tendencies vary across different LLM architectures.

Debiasing strategies can reduce bias propagation in LLMs.

Abstract

The pervasive spread of misinformation and disinformation in social media underscores the critical importance of detecting media bias. While robust Large Language Models (LLMs) have emerged as foundational tools for bias prediction, concerns about inherent biases within these models persist. In this work, we investigate the presence and nature of bias within LLMs and its consequential impact on media bias detection. Departing from conventional approaches that focus solely on bias detection in media content, we delve into biases within the LLM systems themselves. Through meticulous examination, we probe whether LLMs exhibit biases, particularly in political bias prediction and text continuation tasks. Additionally, we explore bias across diverse topics, aiming to uncover nuanced variations in bias expression within the LLM framework. Importantly, we propose debiasing strategies, including prompt engineering and model fine-tuning. Extensive analysis of bias tendencies across different LLMs sheds light on the broader landscape of bias propagation in language models. This study advances our understanding of LLM bias, offering critical insights into its implications for bias detection tasks and paving the way for more robust and equitable AI systems