The RAG Paradox: A Black-Box Attack Exploiting Unintentional Vulnerabilities in Retrieval-Augmented Generation Systems

TL;DR

This paper reveals a structural vulnerability in retrieval-augmented generation systems, where transparency allows attackers to craft poisoned documents that degrade performance while appearing natural, highlighting new security risks.

Contribution

The authors introduce a realistic black-box attack exploiting the RAG paradox, demonstrating how transparency can be used maliciously to undermine RAG system integrity.

Findings

Significantly degrades RAG system performance

Generates natural-looking poisoned documents

Effective in both offline and online settings

Abstract

With the growing adoption of retrieval-augmented generation (RAG) systems, various attack methods have been proposed to degrade their performance. However, most existing approaches rely on unrealistic assumptions in which external attackers have access to internal components such as the retriever. To address this issue, we introduce a realistic black-box attack based on the RAG paradox, a structural vulnerability arising from the system's effort to enhance trust by revealing both the retrieved documents and their sources to users. This transparency enables attackers to observe which sources are used and how information is phrased, allowing them to craft poisoned documents that are more likely to be retrieved and upload them to the identified sources. Moreover, as RAG systems directly provide retrieved content to users, these documents must not only be retrievable but also appear natural and credible to maintain user confidence in the search results. Unlike prior work that focuses solely on improving document retrievability, our attack method explicitly considers both retrievability and user trust in the retrieved content. Both offline and online experiments demonstrate that our method significantly degrades system performance without internal access, while generating natural-looking poisoned documents.