LatestEval: Addressing Data Contamination in Language Model Evaluation through Dynamic and Time-Sensitive Test Construction

TL;DR

LatestEval introduces a dynamic, time-sensitive evaluation method for language models that minimizes data contamination by using recent texts, leading to more accurate assessments of model capabilities.

Contribution

The paper presents an automated pipeline for constructing uncontaminated, recent-text-based reading comprehension evaluations to improve the robustness of language model assessments.

Findings

Models show negligible memorization on LatestEval

LatestEval reduces data contamination in evaluations

Benchmark results indicate more reliable model performance assessment

Abstract

Data contamination in evaluation is getting increasingly prevalent with the emergence of language models pre-trained on super large, automatically crawled corpora. This problem leads to significant challenges in the accurate assessment of model capabilities and generalisations. In this paper, we propose LatestEval, an automatic method that leverages the most recent texts to create uncontaminated reading comprehension evaluations. LatestEval avoids data contamination by only using texts published within a recent time window, ensuring no overlap with the training corpora of pre-trained language models. We develop the LatestEval automated pipeline to 1) gather the latest texts; 2) identify key information, and 3) construct questions targeting the information while removing the existing answers from the context. This encourages models to infer the answers themselves based on the remaining context, rather than just copy-paste. Our experiments demonstrate that language models exhibit negligible memorisation behaviours on LatestEval as opposed to previous benchmarks, suggesting a significantly reduced risk of data contamination and leading to a more robust evaluation. Data and code are publicly available at: https://github.com/liyucheng09/LatestEval.