Effective Code Membership Inference for Code Completion Models via Adversarial Prompts

TL;DR

This paper introduces AdvPrompt-MIA, a novel adversarial prompt-based membership inference attack on code completion models that effectively detects training data membership with high accuracy and transferability.

Contribution

We propose AdvPrompt-MIA, a new method combining adversarial prompts and deep learning to improve membership inference on code models, surpassing existing techniques in accuracy and generalizability.

Findings

Outperforms state-of-the-art baselines with up to 102% AUC gain.

Demonstrates strong transferability across models and datasets.

Effectively captures nuanced memorization patterns in code models.

Abstract

Membership inference attacks (MIAs) on code completion models offer an effective way to assess privacy risks by inferring whether a given code snippet was part of the training data. Existing black- and gray-box MIAs rely on expensive surrogate models or manually crafted heuristic rules, which limit their ability to capture the nuanced memorization patterns exhibited by over-parameterized code language models. To address these challenges, we propose AdvPrompt-MIA, a method specifically designed for code completion models, combining code-specific adversarial perturbations with deep learning. The core novelty of our method lies in designing a series of adversarial prompts that induce variations in the victim code model's output. By comparing these outputs with the ground-truth completion, we construct feature vectors to train a classifier that automatically distinguishes member from non-member samples. This design allows our method to capture richer memorization patterns and accurately infer training set membership. We conduct comprehensive evaluations on widely adopted models, such as Code Llama 7B, over the APPS and HumanEval benchmarks. The results show that our approach consistently outperforms state-of-the-art baselines, with AUC gains of up to 102%. In addition, our method exhibits strong transferability across different models and datasets, underscoring its practical utility and generalizability.