Large Language Models' Detection of Political Orientation in Newspapers

TL;DR

This study evaluates whether large language models reliably assess newspapers' political orientations, revealing significant inconsistencies among models and highlighting the need for improved training and evaluation standards.

Contribution

It systematically compares multiple LLMs' assessments of newspaper political positioning, exposing their inconsistencies and advocating for better training and benchmarking.

Findings

LLMs show high variability in assessing newspaper political orientation.

Different LLMs often disagree on the same newspaper's stance.

There is a need for improved training and community-driven benchmarks.

Abstract

Democratic opinion-forming may be manipulated if newspapers' alignment to political or economical orientation is ambiguous. Various methods have been developed to better understand newspapers' positioning. Recently, the advent of Large Language Models (LLM), and particularly the pre-trained LLM chatbots like ChatGPT or Gemini, hold disruptive potential to assist researchers and citizens alike. However, little is know on whether LLM assessment is trustworthy: do single LLM agrees with experts' assessment, and do different LLMs answer consistently with one another? In this paper, we address specifically the second challenge. We compare how four widely employed LLMs rate the positioning of newspapers, and compare if their answers align with one another. We observe that this is not the case. Over a woldwide dataset, articles in newspapers are positioned strikingly differently by single LLMs, hinting to inconsistent training or excessive randomness in the algorithms. We thus raise a warning when deciding which tools to use, and we call for better training and algorithm development, to cover such significant gap in a highly sensitive matter for democracy and societies worldwide. We also call for community engagement in benchmark evaluation, through our open initiative navai.pro.