Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMs

TL;DR

This paper presents a black-box prompt optimization method using an attacker LLM to reveal higher levels of memorization in victim models, surpassing traditional prompting approaches and exposing training data leakage.

Contribution

Introduces an iterative rejection-sampling prompt optimization technique to uncover memorization in LLMs, highlighting the effectiveness of instruction-based prompts and automated attack avenues.

Findings

Instruction-tuned models can leak training data as much as base models.

Contexts beyond training data can cause data leakage.

Using other LLMs' instructions enables automated memorization attacks.

Abstract

In this paper, we introduce a black-box prompt optimization method that uses an attacker LLM agent to uncover higher levels of memorization in a victim agent, compared to what is revealed by prompting the target model with the training data directly, which is the dominant approach of quantifying memorization in LLMs. We use an iterative rejection-sampling optimization process to find instruction-based prompts with two main characteristics: (1) minimal overlap with the training data to avoid presenting the solution directly to the model, and (2) maximal overlap between the victim model's output and the training data, aiming to induce the victim to spit out training data. We observe that our instruction-based prompts generate outputs with 23.7% higher overlap with training data compared to the baseline prefix-suffix measurements. Our findings show that (1) instruction-tuned models can expose pre-training data as much as their base-models, if not more so, (2) contexts other than the original training data can lead to leakage, and (3) using instructions proposed by other LLMs can open a new avenue of automated attacks that we should further study and explore. The code can be found at https://github.com/Alymostafa/Instruction_based_attack .