Training on the Benchmark Is Not All You Need

TL;DR

This paper introduces a simple, effective method to detect data leakage in large language models' pre-training datasets by analyzing model responses to shuffled multiple-choice options, revealing significant leakage in popular open-source LLMs.

Contribution

The paper proposes a novel data leakage detection technique based on log probability analysis of shuffled options, applicable under gray-box conditions, and evaluates leakage levels across multiple models and benchmarks.

Findings

Effective detection of data leakage in LLMs' pre-training data.

Qwen LLMs exhibit the highest data leakage among tested models.

Method works without access to training data or model weights.

Abstract

The success of Large Language Models (LLMs) relies heavily on the huge amount of pre-training data learned in the pre-training phase. The opacity of the pre-training process and the training data causes the results of many benchmark tests to become unreliable. If any model has been trained on a benchmark test set, it can seriously hinder the health of the field. In order to automate and efficiently test the capabilities of large language models, numerous mainstream benchmarks adopt a multiple-choice format. As the swapping of the contents of multiple-choice options does not affect the meaning of the question itself, we propose a simple and effective data leakage detection method based on this property. Specifically, we shuffle the contents of the options in the data to generate the corresponding derived data sets, and then detect data leakage based on the model's log probability distribution over the derived data sets. If there is a maximum and outlier in the set of log probabilities, it indicates that the data is leaked. Our method is able to work under gray-box conditions without access to model training data or weights, effectively identifying data leakage from benchmark test sets in model pre-training data, including both normal scenarios and complex scenarios where options may have been shuffled intentionally or unintentionally. Through experiments based on two LLMs and benchmark designs, we demonstrate the effectiveness of our method. In addition, we evaluate the degree of data leakage of 35 mainstream open-source LLMs on four benchmark datasets and give a ranking of the leaked LLMs for each benchmark, and we find that the Qwen family of LLMs has the highest degree of data leakage.