RAG Safety: Exploring Knowledge Poisoning Attacks to Retrieval-Augmented Generation

TL;DR

This paper investigates the security vulnerabilities of knowledge graph-based retrieval-augmented generation (KG-RAG) systems, revealing their susceptibility to data poisoning attacks that can mislead AI responses by injecting malicious knowledge.

Contribution

It presents the first systematic study of data poisoning attacks on KG-RAG, proposing a practical attack method and demonstrating its effectiveness across multiple benchmarks and models.

Findings

KG-RAG systems are vulnerable to targeted data poisoning attacks.

Minimal perturbations in knowledge graphs can significantly degrade RAG performance.

The study highlights safety risks and robustness challenges in KG-RAG systems.

Abstract

Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by retrieving external data to mitigate hallucinations and outdated knowledge issues. Benefiting from the strong ability in facilitating diverse data sources and supporting faithful reasoning, knowledge graphs (KGs) have been increasingly adopted in RAG systems, giving rise to KG-based RAG (KG-RAG) methods. Though RAG systems are widely applied in various applications, recent studies have also revealed its vulnerabilities to data poisoning attacks, where malicious information injected into external knowledge sources can mislead the system into producing incorrect or harmful responses. However, these studies focus exclusively on RAG systems using unstructured textual data sources, leaving the security risks of KG-RAG largely unexplored, despite the fact that KGs present unique vulnerabilities due to their structured and editable nature. In this work, we conduct the first systematic investigation of the security issue of KG-RAG methods through data poisoning attacks. To this end, we introduce a practical, stealthy attack setting that aligns with real-world implementation. We propose an attack strategy that first identifies adversarial target answers and then inserts perturbation triples to complete misleading inference chains in the KG, increasing the likelihood that KG-RAG methods retrieve and rely on these perturbations during generation. Through extensive experiments on two benchmarks and four recent KG-RAG methods, our attack strategy demonstrates strong effectiveness in degrading KG-RAG performance, even with minimal KG perturbations. In-depth analyses are also conducted to understand the safety threats within the internal stages of KG-RAG systems and to explore the robustness of LLMs against adversarial knowledge.