Information Retrieval Induced Safety Degradation in AI Agents

TL;DR

This paper demonstrates that enabling external retrieval in AI agents can lead to safety degradation, increasing bias, harmful content, and unsafe behaviors despite retrieval accuracy or mitigation efforts.

Contribution

It reveals the counterintuitive safety risks of retrieval-enabled AI agents and highlights the need for new mitigation strategies to maintain safety and fairness.

Findings

Retrieval access reduces refusal rates but increases bias and harmful content.

Retrieval-enabled models often behave more unsafely than uncensored models.

Safety degradation persists even with mitigation and high retrieval accuracy.

Abstract

Despite the growing integration of retrieval-enabled AI agents into society, their safety and ethical behavior remain inadequately understood. In particular, the integration of LLMs and AI agents with external information sources and real-world environments raises critical questions about how they engage with and are influenced by these external data sources and interactive contexts. This study investigates how expanding retrieval access -- from no external sources to Wikipedia-based retrieval and open web search -- affects model reliability, bias propagation, and harmful content generation. Through extensive benchmarking of censored and uncensored LLMs and AI agents, our findings reveal a consistent degradation in refusal rates, bias sensitivity, and harmfulness safeguards as models gain broader access to external sources, culminating in a phenomenon we term safety degradation. Notably, retrieval-enabled agents built on aligned LLMs often behave more unsafely than uncensored models without retrieval. This effect persists even under strong retrieval accuracy and prompt-based mitigation, suggesting that the mere presence of retrieved content reshapes model behavior in structurally unsafe ways. These findings underscore the need for robust mitigation strategies to ensure fairness and reliability in retrieval-enabled and increasingly autonomous AI systems.