The Impact of Post-training on Data Contamination

TL;DR

This study investigates how dataset contamination affects large language models during post-training, revealing that contamination causes performance inflation which can be mitigated through specific post-training methods and scale considerations.

Contribution

It provides a controlled analysis of contamination effects in large language models and compares the impacts of supervised fine-tuning and reinforcement learning post-training methods.

Findings

Contamination causes performance spikes that diminish with continued pre-training.

SFT inflates scores only on contaminated tasks, while GRPO inflates on both contaminated and uncontaminated tasks.

Larger models memorize more and translate leakage into more general capabilities.

Abstract

We present a controlled study of how dataset contamination interacts with the post-training stages now standard in large language model training pipelines. Starting from clean checkpoints of Qwen2.5 (0.5B/1.5B) and Gemma3 (1B/4B), we inject five copies of GSM8K and MBPP test items into the first 2B tokens of an otherwise 25B token extended pre-training dataset. We then compare the contaminated and clean models both immediately after pre-training and again after two popular post-training methods: supervised fine-tuning (SFT) and reinforcement learning (RL) with group relative policy optimization (GRPO). The applied post-training steps do not have any contamination. Across math and coding benchmarks, we find three consistent patterns: (i) Contamination causes performance spikes that are gradually diminished with continued pre-training. After even 25B tokens the apparent performance inflation of contamination can become close to zero. (ii) Both SFT and GRPO resurface the leaked information, but with different external validity: SFT inflates scores only on the contaminated tasks, whereas GRPO also inflates performance on uncontaminated counterparts (GSMPlus, HumanEval). (iii) Model scale amplifies these tendencies, larger Supervised Fine Tuned models memorize more, while larger GRPO models translate leakage into more generalizable capabilities. Our results underscore the need for contamination audits \emph{after} post-training and suggest that RL-based post-training, although not immune, can help alleviate contamination-related over-estimation problems.