Addressing Corpus Knowledge Poisoning Attacks on RAG Using Sparse Attention

TL;DR

This paper introduces SDAG, a sparse attention mechanism that prevents harmful cross-document interactions in RAG, significantly improving robustness against corpus knowledge poisoning attacks without requiring fine-tuning.

Contribution

The paper proposes SDAG, a novel block-sparse attention method for RAG that enhances resistance to poisoning attacks with minimal inference-time modifications.

Findings

SDAG substantially reduces attack success rates in RAG-based QA.

SDAG outperforms standard causal attention in defending against corpus poisoning.

Combining SDAG with existing defenses yields statistically significant improvements.

Abstract

Retrieval Augmented Generation (RAG) is a highly effective paradigm for keeping LLM-based responses up-to-date and reducing the likelihood of hallucinations. Yet, RAG was recently shown to be quite vulnerable to corpus knowledge poisoning: an attacker injects misleading documents to the corpus to steer an LLM's output to an undesired response. We argue that the standard causal attention mechanism in LLMs enables harmful cross-document interactions, specifically in cases of attacks. Accordingly, we introduce a novel defense approach for RAG: Sparse Document Attention RAG (SDAG). This is a block-sparse attention mechanism that disallows cross-attention between retrieved documents. SDAG requires a minimal inference-time change to the attention mask; furthermore, no fine-tuning or additional architectural changes are needed. We present an empirical evaluation of LLM-based question answering (QA) with a variety of attack strategies on RAG. We show that our SDAG method substantially outperforms the standard causal attention mechanism in terms of attack success rate. We further demonstrate the clear merits of integrating SDAG with state-of-the-art RAG defense methods. Specifically, the integration results in performance that is statistically significantly better than the state-of-the-art.