Safeguarding Privacy of Retrieval Data against Membership Inference Attacks: Is This Query Too Close to Home?

TL;DR

This paper presents a similarity-based detection framework for membership inference attacks in retrieval-augmented generation systems, effectively protecting private data while maintaining utility and system-agnosticism.

Contribution

It introduces a novel similarity-based detection method and a detect-and-hide strategy that obfuscates attackers without compromising system utility.

Findings

Successfully detects state-of-the-art MIAs

Effectively obfuscates attackers in RAG systems

Maintains data utility and system-agnosticism

Abstract

Retrieval-augmented generation (RAG) mitigates the hallucination problem in large language models (LLMs) and has proven effective for personalized usages. However, delivering private retrieved documents directly to LLMs introduces vulnerability to membership inference attacks (MIAs), which try to determine whether the target data point exists in the private external database or not. Based on the insight that MIA queries typically exhibit high similarity to only one target document, we introduce a novel similarity-based MIA detection framework designed for the RAG system. With the proposed method, we show that a simple detect-and-hide strategy can successfully obfuscate attackers, maintain data utility, and remain system-agnostic against MIA. We experimentally prove its detection and defense against various state-of-the-art MIA methods and its adaptability to existing RAG systems.