An evaluation of LLMs for political bias in Western media: Israel-Hamas and Ukraine-Russia wars

TL;DR

This study evaluates how large language models detect political bias in Western media coverage of the Russia-Ukraine and Israel-Hamas conflicts, revealing biases vary by media outlet, conflict, and LLM used.

Contribution

It introduces a comparative analysis of multiple LLMs in assessing political bias in media reporting during major conflicts, highlighting the influence of model architecture and training data.

Findings

DeepSeek shows consistent left-leaning bias

BERT and Gemini are closer to centrist positions

Media bias shifts are more pronounced during Israel-Hamas conflict

Abstract

Political bias in media plays a critical role in shaping public opinion, voter behaviour, and broader democratic discourse. Subjective opinions and political bias can be found in media sources, such as newspapers, depending on their funding mechanisms and alliances with political parties. Automating the detection of political biases in media content can limit biases in elections. The impact of large language models (LLMs) in politics and media studies is becoming prominent. In this study, we utilise LLMs to compare the left-wing, right-wing, and neutral political opinions expressed in the Guardian and BBC. We review newspaper reporting that includes significant events such as the Russia-Ukraine war and the Hamas-Israel conflict. We analyse the proportion for each opinion to find the bias under different LLMs, including BERT, Gemini, and DeepSeek. Our results show that after the outbreak of the wars, the political bias of Western media shifts towards the left-wing and each LLM gives a different result. DeepSeek consistently showed a stable Left-leaning tendency, while BERT and Gemini remained closer to the Centre. The BBC and The Guardian showed distinct reporting behaviours across the two conflicts. In the Russia-Ukraine war, both outlets maintained relatively stable positions; however, in the Israel-Hamas conflict, we identified larger political bias shifts, particularly in Guardian coverage, suggesting a more event-driven pattern of reporting bias. These variations suggest that LLMs are shaped not only by their training data and architecture, but also by underlying worldviews with associated political biases.