DICE: Detecting In-distribution Contamination in LLM's Fine-tuning Phase for Math Reasoning

TL;DR

This paper introduces DICE, a novel method that uses internal states of large language models to detect in-distribution contamination during fine-tuning, helping ensure accurate evaluation of LLMs on math reasoning benchmarks.

Contribution

DICE is the first approach to identify in-distribution contamination by analyzing internal model states, improving detection accuracy and generalization across multiple benchmarks.

Findings

DICE achieves high accuracy in detecting contamination across various LLMs.

DICE generalizes well to multiple benchmarks with similar distributions.

DICE's predictions correlate strongly with LLM fine-tuning performance.

Abstract

The advancement of large language models (LLMs) relies on evaluation using public benchmarks, but data contamination can lead to overestimated performance. Previous researches focus on detecting contamination by determining whether the model has seen the exact same data during training. Besides, prior work has already shown that even training on data similar to benchmark data inflates performance, namely \emph{In-distribution contamination}. In this work, we argue that in-distribution contamination can lead to the performance drop on OOD benchmarks. To effectively detect in-distribution contamination, we propose DICE, a novel method that leverages the internal states of LLMs to locate-then-detect the contamination. DICE first identifies the most sensitive layer to contamination, then trains a classifier based on the internal states of that layer. Experiments reveal DICE's high accuracy in detecting in-distribution contamination across various LLMs and math reasoning datasets. We also show the generalization capability of the trained DICE detector, which is able to detect contamination across multiple benchmarks with similar distributions. Additionally, we find that DICE's predictions correlate with the performance of LLMs fine-tuned by either us or other organizations, achieving a coefficient of determination ($R^2$) between 0.61 and 0.75. The code and data are available at https://github.com/THU-KEG/DICE.