DCR: Quantifying Data Contamination in LLMs Evaluation

TL;DR

This paper presents the DCR framework, a lightweight and interpretable method to detect and quantify data contamination in LLM evaluations, improving the reliability of benchmark performance assessments.

Contribution

The paper introduces the DCR framework, a novel, efficient pipeline that quantifies data contamination risk at multiple levels, enhancing fairness in LLM benchmarking.

Findings

DCR accurately diagnoses contamination severity across models and tasks.

Adjusted performance metrics with DCR reduce bias caused by data contamination.

Framework demonstrates high reliability with less than 4% average error.

Abstract

The rapid advancement of large language models (LLMs) has heightened concerns about benchmark data contamination (BDC), where models inadvertently memorize evaluation data during the training process, inflating performance metrics, and undermining genuine generalization assessment. This paper introduces the Data Contamination Risk (DCR) framework, a lightweight, interpretable pipeline designed to detect and quantify BDC risk across four granular levels: semantic, informational, data, and label. By synthesizing contamination scores via a fuzzy inference system, DCR produces a unified DCR Factor that adjusts raw accuracy to reflect contamination-aware performance. Validated on 9 LLMs (0.5B-72B) across sentiment analysis, fake news detection, and arithmetic reasoning tasks, the DCR framework reliably diagnoses contamination severity and with accuracy adjusted using the DCR Factor to within 4% average error across the three benchmarks compared to the uncontaminated baseline. Emphasizing computational efficiency and transparency, DCR provides a practical tool for integrating contamination assessment into routine evaluations, fostering fairer comparisons and enhancing the credibility of LLM benchmarking practices.