LLMs left, right, and center: Assessing GPT's capabilities to label political bias from web domains

TL;DR

This study evaluates GPT-4's ability to classify political bias of news sources from URLs, finding high correlation with human ratings but noting limitations in abstention rates and polarization biases.

Contribution

It demonstrates GPT-4's potential as a scalable tool for political bias classification, highlighting its strengths and limitations compared to human assessments.

Findings

High correlation ($\rho = .89$) with human bias ratings

GPT-4 abstains from about two-thirds of classifications

Tends to produce more polarized and slightly left-leaning outputs

Abstract

This research investigates whether OpenAI's GPT-4, a state-of-the-art large language model, can accurately classify the political bias of news sources based solely on their URLs. Given the subjective nature of political labels, third-party bias ratings like those from Ad Fontes Media, AllSides, and Media Bias/Fact Check (MBFC) are often used in research to analyze news source diversity. This study aims to determine if GPT-4 can replicate these human ratings on a seven-degree scale ("far-left" to "far-right"). The analysis compares GPT-4's classifications against MBFC's, and controls for website popularity using Open PageRank scores. Findings reveal a high correlation ($\text{Spearman's } \rho = .89$, $n = 5,877$, $p < 0.001$) between GPT-4's and MBFC's ratings, indicating the model's potential reliability. However, GPT-4 abstained from classifying approximately $\frac{2}{3}$ of the dataset. It is more likely to abstain from rating unpopular websites, which also suffer from less accurate assessments. The LLM tends to avoid classifying sources that MBFC considers to be centrist, resulting in more polarized outputs. Finally, this analysis shows a slight leftward skew in GPT's classifications compared to MBFC's. Therefore, while this paper suggests that while GPT-4 can be a scalable, cost-effective tool for political bias classification of news websites, its use should be as a complement to human judgment to mitigate biases.