Dataset Protection via Watermarked Canaries in Retrieval-Augmented LLMs

TL;DR

This paper proposes a novel watermarking method using watermarked canary documents to protect dataset ownership in Retrieval-Augmented LLMs, enabling effective detection of unauthorized dataset usage without impairing system performance.

Contribution

It introduces a new watermarking approach with synthetic canary documents for dataset protection in RAG systems, ensuring stealthiness, detectability, and minimal data perturbation.

Findings

High query efficiency in detecting unauthorized use

Watermarked canaries are stealthy and statistically provable

Minimal impact on RAG system performance

Abstract

Retrieval-Augmented Generation (RAG) has become an effective method for enhancing large language models (LLMs) with up-to-date knowledge. However, it poses a significant risk of IP infringement, as IP datasets may be incorporated into the knowledge database by malicious Retrieval-Augmented LLMs (RA-LLMs) without authorization. To protect the rights of the dataset owner, an effective dataset membership inference algorithm for RA-LLMs is needed. In this work, we introduce a novel approach to safeguard the ownership of text datasets and effectively detect unauthorized use by the RA-LLMs. Our approach preserves the original data completely unchanged while protecting it by inserting specifically designed canary documents into the IP dataset. These canary documents are created with synthetic content and embedded watermarks to ensure uniqueness, stealthiness, and statistical provability. During the detection process, unauthorized usage is identified by querying the canary documents and analyzing the responses of RA-LLMs for statistical evidence of the embedded watermark. Our experimental results demonstrate high query efficiency, detectability, and stealthiness, along with minimal perturbation to the original dataset, all without compromising the performance of the RAG system.