In Agents We Trust, but Who Do Agents Trust? Latent Source Preferences Steer LLM Generations

TL;DR

This paper investigates how large language model (LLM) agents display systematic source preferences when selecting information, revealing biases that influence what users see and highlighting the need for transparency and control mechanisms.

Contribution

It uncovers latent source preferences in LLMs, demonstrating their consistency, sensitivity to context, and impact on information presentation, which was previously underexplored.

Findings

LLMs exhibit strong, predictable source preferences.

Preferences are influenced by contextual framing.

Preferences persist despite explicit prompts to avoid them.

Abstract

Agents based on Large Language Models (LLMs) are increasingly being deployed as interfaces to information on online platforms. These agents filter, prioritize, and synthesize information retrieved from the platforms' back-end databases or via web search. In these scenarios, LLM agents govern the information users receive, by drawing users' attention to particular instances of retrieved information at the expense of others. While much prior work has focused on biases in the information LLMs themselves generate, less attention has been paid to the factors that influence what information LLMs select and present to users. We hypothesize that when information is attributed to specific sources (e.g., particular publishers, journals, or platforms), current LLMs exhibit systematic latent source preferences- that is, they prioritize information from some sources over others. Through controlled experiments on twelve LLMs from six model providers, spanning both synthetic and real-world tasks, we find that several models consistently exhibit strong and predictable source preferences. These preferences are sensitive to contextual framing, can outweigh the influence of content itself, and persist despite explicit prompting to avoid them. They also help explain phenomena such as the observed left-leaning skew in news recommendations in prior work. Our findings advocate for deeper investigation into the origins of these preferences, as well as for mechanisms that provide users with transparency and control over the biases guiding LLM-powered agents.