Quantifying the Effect of Test Set Contamination on Generative Evaluations

TL;DR

This paper investigates how test set contamination affects the evaluation of generative language models, revealing that even minimal contamination can significantly skew performance metrics and influence memorization during inference.

Contribution

It provides a comprehensive quantitative analysis of test set contamination effects on generative evaluations, including new insights into model scaling, overtraining, and inference factors.

Findings

Contamination improves performance with larger models and more contaminated data.

Single test set replica can lead to performance below the irreducible error.

High sampling temperatures and shorter outputs reduce memorization effects.

Abstract

As frontier AI systems are pretrained on web-scale data, test set contamination has become a critical concern for accurately assessing their capabilities. While research has thoroughly investigated the impact of test set contamination on discriminative evaluations like multiple-choice question-answering, comparatively little research has studied the impact of test set contamination on generative evaluations. In this work, we quantitatively assess the effect of test set contamination on generative evaluations through the language model lifecycle. We pretrain language models on mixtures of web data and the MATH benchmark, sweeping model sizes and number of test set replicas contaminating the pretraining corpus; performance improves with contamination and model size. Using scaling laws, we make a surprising discovery: including even a single test set replica enables models to achieve lower loss than the irreducible error of training on the uncontaminated corpus. We then study further training: overtraining with fresh data reduces the effects of contamination, whereas supervised finetuning on the training set can either increase or decrease performance on test data, depending on the amount of pretraining contamination. Finally, at inference, we identify factors that modulate memorization: high sampling temperatures mitigate contamination effects, and longer solutions are exponentially more difficult to memorize than shorter ones, presenting a contrast with discriminative evaluations, where solutions are only a few tokens in length. By characterizing how generation and memorization interact, we highlight a new layer of complexity for trustworthy evaluation of AI systems.