The Democratic Paradox in Large Language Models' Underestimation of Press Freedom

TL;DR

This paper reveals systematic biases in large language models' assessments of press freedom, showing they underestimate global press freedom, especially in countries with strong press freedom, and display home bias, which could distort public understanding.

Contribution

It uncovers three systematic distortions in LLM evaluations of press freedom, highlighting underestimation, differential misalignment, and home bias, which are novel findings in the context of AI and democratic institutions.

Findings

LLMs underestimate press freedom in 71-93% of countries.

LLMs disproportionately underestimate press freedom where it is strongest.

Most LLMs show positive home bias, rating their own country's press freedom more favorably.

Abstract

As Large Language Models (LLMs) increasingly mediate global information access for millions of users worldwide, their alignment and biases have the potential to shape public understanding and trust in fundamental democratic institutions, such as press freedom. In this study, we uncover three systematic distortions in the way six popular LLMs evaluate press freedom in 180 countries compared to expert assessments of the World Press Freedom Index (WPFI). The six LLMs exhibit a negative misalignment, consistently underestimating press freedom, with individual models rating between 71% to 93% of countries as less free. We also identify a paradoxical pattern we term differential misalignment: LLMs disproportionately underestimate press freedom in countries where it is strongest. Additionally, five of the six LLMs exhibit positive home bias, rating their home countries' press freedoms more favorably than would be expected given their negative misalignment with the human benchmark. In some cases, LLMs rate their home countries between 7% to 260% more positively than expected. If LLMs are set to become the next search engines and some of the most important cultural tools of our time, they must ensure accurate representations of the state of our human and civic rights globally.