BiasLab: Toward Explainable Political Bias Detection with Dual-Axis Annotations and Rationale Indicators

TL;DR

BiasLab introduces a new dataset of 300 political news articles with dual-axis bias annotations and rationale indicators, enabling explainable political bias detection and analysis of perception drift using human and GPT-4o annotations.

Contribution

This work provides a novel dataset with rich annotations and rationale indicators for political bias, along with baseline models and analysis of annotation methods including GPT-4o simulation.

Findings

Inter-annotator agreement quantification

Misalignment analysis with source outlet bias

GPT-4o mirrors human annotation asymmetries

Abstract

We present BiasLab, a dataset of 300 political news articles annotated for perceived ideological bias. These articles were selected from a curated 900-document pool covering diverse political events and source biases. Each article is labeled by crowdworkers along two independent scales, assessing sentiment toward the Democratic and Republican parties, and enriched with rationale indicators. The annotation pipeline incorporates targeted worker qualification and was refined through pilot-phase analysis. We quantify inter-annotator agreement, analyze misalignment with source-level outlet bias, and organize the resulting labels into interpretable subsets. Additionally, we simulate annotation using schema-constrained GPT-4o, enabling direct comparison to human labels and revealing mirrored asymmetries, especially in misclassifying subtly right-leaning content. We define two modeling tasks: perception drift prediction and rationale type classification, and report baseline performance to illustrate the challenge of explainable bias detection. BiasLab's rich rationale annotations provide actionable interpretations that facilitate explainable modeling of political bias, supporting the development of transparent, socially aware NLP systems. We release the dataset, annotation schema, and modeling code to encourage research on human-in-the-loop interpretability and the evaluation of explanation effectiveness in real-world settings.