Detecting Data Contamination in LLMs via In-Context Learning

TL;DR

The paper introduces CoDeC, a practical method to detect and quantify training data contamination in large language models by analyzing in-context learning effects on model confidence.

Contribution

It presents a simple, automated, and model-agnostic contamination detection method that distinguishes between memorized and unseen data in large language models.

Findings

CoDeC produces interpretable contamination scores separating seen and unseen datasets.

In-context examples boost confidence for unseen data but may reduce it for training data.

Strong evidence of memorization found in open-weight models with undisclosed training data.

Abstract

We present Contamination Detection via Context (CoDeC), a practical and accurate method to detect and quantify training data contamination in large language models. CoDeC distinguishes between data memorized during training and data outside the training distribution by measuring how in-context learning affects model performance. We find that in-context examples typically boost confidence for unseen datasets but may reduce it when the dataset was part of training, due to disrupted memorization patterns. Experiments show that CoDeC produces interpretable contamination scores that clearly separate seen and unseen datasets, and reveals strong evidence of memorization in open-weight models with undisclosed training corpora. The method is simple, automated, and both model- and dataset-agnostic, making it easy to integrate with benchmark evaluations.