How Much Can We Forget about Data Contamination?

TL;DR

This paper investigates the impact of data contamination in large language model training, showing that modern models can forget contaminated data over time and that overfitting due to small-scale contamination is less severe than previously thought.

Contribution

It provides a quantitative analysis of data contamination effects and demonstrates that large-scale training can mitigate overfitting from benchmark data leakage.

Findings

Minor contamination leads to overfitting if data is small-scale.

Modern LLMs can forget contaminated data with sufficient training.

Weight decay influences forgetting faster than expected.

Abstract

The leakage of benchmark data into the training data has emerged as a significant challenge for evaluating the capabilities of large language models (LLMs). In this work, we challenge the common assumption that small-scale contamination renders benchmark evaluations invalid. First, we experimentally quantify the magnitude of benchmark overfitting based on scaling along three dimensions: The number of model parameters (up to 1.6B), the number of times an example is seen (up to 144), and the number of training tokens (up to 40B). If model and data follow the Chinchilla scaling laws, minor contamination indeed leads to overfitting. At the same time, even 144 times of contamination can be forgotten if the training data is scaled beyond five times Chinchilla, a regime characteristic of many modern LLMs. Continual pre-training of OLMo-7B corroborates these results. Next, we study the impact of the weight decay parameter on example forgetting, showing that empirical forgetting occurs faster than the cumulative weight decay. This allows us to gauge the degree of example forgetting in large-scale training runs, indicating that many LLMs, including Lllama 3 405B, have forgotten the data seen at the beginning of training.