ConfusedPilot: Confused Deputy Risks in RAG-based LLMs

TL;DR

This paper identifies security vulnerabilities in retrieval augmented generation (RAG) systems, demonstrating how they can be exploited to corrupt responses, leak sensitive data, and propagate misinformation, with implications for enterprise security.

Contribution

It introduces ConfusedPilot, a new class of security vulnerabilities in RAG systems, and provides analysis and guidelines to mitigate these risks.

Findings

Malicious text can corrupt RAG responses.

Cache mechanisms can leak secret data.

Vulnerabilities can propagate misinformation.

Abstract

Retrieval augmented generation (RAG) is a process where a large language model (LLM) retrieves useful information from a database and then generates the responses. It is becoming popular in enterprise settings for daily business operations. For example, Copilot for Microsoft 365 has accumulated millions of businesses. However, the security implications of adopting such RAG-based systems are unclear. In this paper, we introduce ConfusedPilot, a class of security vulnerabilities of RAG systems that confuse Copilot and cause integrity and confidentiality violations in its responses. First, we investigate a vulnerability that embeds malicious text in the modified prompt in RAG, corrupting the responses generated by the LLM. Second, we demonstrate a vulnerability that leaks secret data, which leverages the caching mechanism during retrieval. Third, we investigate how both vulnerabilities can be exploited to propagate misinformation within the enterprise and ultimately impact its operations, such as sales and manufacturing. We also discuss the root cause of these attacks by investigating the architecture of a RAG-based system. This study highlights the security vulnerabilities in today's RAG-based systems and proposes design guidelines to secure future RAG-based systems.