Con-ReCall: Detecting Pre-training Data in LLMs via Contrastive Decoding

TL;DR

Con-ReCall introduces a contrastive decoding method to detect pre-training data in large language models by leveraging subtle distributional shifts between member and non-member contexts, improving privacy risk assessment.

Contribution

It presents a novel contrastive decoding approach that effectively exploits distributional differences for membership inference in LLMs, surpassing existing methods.

Findings

Achieves state-of-the-art results on WikiMIA benchmark

Robust against various text manipulation techniques

Enhances privacy risk detection in LLMs

Abstract

The training data in large language models is key to their success, but it also presents privacy and security risks, as it may contain sensitive information. Detecting pre-training data is crucial for mitigating these concerns. Existing methods typically analyze target text in isolation or solely with non-member contexts, overlooking potential insights from simultaneously considering both member and non-member contexts. While previous work suggested that member contexts provide little information due to the minor distributional shift they induce, our analysis reveals that these subtle shifts can be effectively leveraged when contrasted with non-member contexts. In this paper, we propose Con-ReCall, a novel approach that leverages the asymmetric distributional shifts induced by member and non-member contexts through contrastive decoding, amplifying subtle differences to enhance membership inference. Extensive empirical evaluations demonstrate that Con-ReCall achieves state-of-the-art performance on the WikiMIA benchmark and is robust against various text manipulation techniques.