Through the Stealth Lens: Rethinking Attacks and Defenses in RAG

TL;DR

This paper investigates the stealthiness of poisoning attacks on retrieval-augmented generation systems, formalizes detection challenges, and proposes an attention-based filtering method that improves defense effectiveness against such attacks.

Contribution

It introduces a formal framework for stealth in RAG attacks, and develops an attention-based filtering technique to detect and mitigate poisoned passages.

Findings

Attention-based filtering improves defense accuracy by up to 20%.

Stealthier adaptive attacks can achieve up to 35% success rate.

Formalization of stealth using a distinguishability-based security game.

Abstract

Retrieval-augmented generation (RAG) systems are vulnerable to attacks that inject poisoned passages into the retrieved set, even at low corruption rates. We show that existing attacks are not designed to be stealthy, allowing reliable detection and mitigation. We formalize stealth using a distinguishability-based security game. If a few poisoned passages are designed to control the response, they must differentiate themselves from benign ones, inherently compromising stealth. This motivates the need for attackers to rigorously analyze intermediate signals involved in generation$\unicode{x2014}$such as attention patterns or next-token probability distributions$\unicode{x2014}$to avoid easily detectable traces of manipulation. Leveraging attention patterns, we propose a passage-level score$\unicode{x2014}$the Normalized Passage Attention Score$\unicode{x2014}$used by our Attention-Variance Filter algorithm to identify and filter potentially poisoned passages. This method mitigates existing attacks, improving accuracy by up to $\sim 20 \%$ over baseline defenses. To probe the limits of attention-based defenses, we craft stealthier adaptive attacks that obscure such traces, achieving up to $35 \%$ attack success rate, and highlight the challenges in improving stealth.