Confundo: Learning to Generate Robust Poison for Practical RAG Systems

TL;DR

Confundo introduces a learning-based poisoning framework that enhances attack effectiveness and stealthiness in practical RAG systems, revealing significant security vulnerabilities and proposing defenses against content poisoning.

Contribution

The paper presents Confundo, a novel learning-to-poison framework that outperforms existing attacks by addressing real-world processing and query variability in RAG systems.

Findings

Confundo achieves higher attack success rates across datasets.

It remains effective even with existing defenses.

The framework can manipulate factuality, bias, and hallucinations.

Abstract

Retrieval-augmented generation (RAG) is increasingly deployed in real-world applications, where its reference-grounded design makes outputs appear trustworthy. This trust has spurred research on poisoning attacks that craft malicious content, inject it into knowledge sources, and manipulate RAG responses. However, when evaluated in practical RAG systems, existing attacks suffer from severely degraded effectiveness. This gap stems from two overlooked realities: (i) content is often processed before use, which can fragment the poison and weaken its effect, and (ii) users often do not issue the exact queries anticipated during attack design. These factors can lead practitioners to underestimate risks and develop a false sense of security. To better characterize the threat to practical systems, we present Confundo, a learning-to-poison framework that fine-tunes a large language model as a poison generator to achieve high effectiveness, robustness, and stealthiness. Confundo provides a unified framework supporting multiple attack objectives, demonstrated by manipulating factual correctness, inducing biased opinions, and triggering hallucinations. By addressing these overlooked challenges, Confundo consistently outperforms a wide range of purpose-built attacks across datasets and RAG configurations by large margins, even in the presence of defenses. Beyond exposing vulnerabilities, we also present a defensive use case that protects web content from unauthorized incorporation into RAG systems via scraping, with no impact on user experience.