Towards Unsupervised Adversarial Document Detection in Retrieval Augmented Generation Systems

TL;DR

This paper introduces an unsupervised method for detecting adversarial and manipulated context documents in retrieval augmented generation systems, enhancing security without needing labeled attack data.

Contribution

It proposes a novel unsupervised detection approach using generator activations, embeddings, and entropy measures, capable of identifying zero-day attacks in RAG systems.

Findings

Unsupervised indicators effectively detect adversarial contexts.

Elementary statistical outlier detection shows promising results.

Target prompt information is not necessary for detection.

Abstract

Retrieval augmented generation systems have become an integral part of everyday life. Whether in internet search engines, email systems, or service chatbots, these systems are based on context retrieval and answer generation with large language models. With their spread, also the security vulnerabilities increase. Attackers become increasingly focused on these systems and various hacking approaches are developed. Manipulating the context documents is a way to persist attacks and make them affect all users. Therefore, detecting compromised, adversarial context documents early is crucial for security. While supervised approaches require a large amount of labeled adversarial contexts, we propose an unsupervised approach, being able to detect also zero day attacks. We conduct a preliminary study to show appropriate indicators for adversarial contexts. For that purpose generator activations, output embeddings, and an entropy-based uncertainty measure turn out as suitable, complementary quantities. With an elementary statistical outlier detection, we propose and compare their detection abilities. Furthermore, we show that the target prompt, which the attacker wants to manipulate, is not required for a successful detection. Moreover, our results indicate that a simple context summary generation might even be superior in finding manipulated contexts.