When Neutral Summaries are not that Neutral: Quantifying Political Neutrality in LLM-Generated News Summaries

TL;DR

This study assesses the political neutrality of large language models in summarizing polarizing US news topics, revealing biases towards Democratic perspectives and vocabulary convergence in their outputs.

Contribution

It introduces a novel method to quantify political bias in LLM-generated summaries across multiple contentious topics using a large news corpus.

Findings

LLMs show a consistent pro-Democratic bias in summaries.

Gun control and healthcare are the most biased topics.

High vocabulary overlap indicates convergence in political representations.

Abstract

In an era where societal narratives are increasingly shaped by algorithmic curation, investigating the political neutrality of LLMs is an important research question. This study presents a fresh perspective on quantifying the political neutrality of LLMs through the lens of abstractive text summarization of polarizing news articles. We consider five pressing issues in current US politics: abortion, gun control/rights, healthcare, immigration, and LGBTQ+ rights. Via a substantial corpus of 20,344 news articles, our study reveals a consistent trend towards pro-Democratic biases in several well-known LLMs, with gun control and healthcare exhibiting the most pronounced biases (max polarization differences of -9.49% and -6.14%, respectively). Further analysis uncovers a strong convergence in the vocabulary of the LLM outputs for these divisive topics (55% overlap for Democrat-leaning representations, 52% for Republican). Being months away from a US election of consequence, we consider our findings important.