VarBench: Robust Language Model Benchmarking Through Dynamic Variable Perturbation

TL;DR

This paper introduces VarBench, a dynamic benchmarking method that variabilizes test cases to provide fair, contamination-resistant evaluation of large language models across multiple datasets.

Contribution

It proposes a novel variable perturbation approach for benchmarking, enabling dynamic, fair, and contamination-resistant evaluation of language models.

Findings

Improved accuracy in assessing true model capabilities.

Effective mitigation of data contamination issues.

Versatile application across diverse datasets.

Abstract

As large language models achieve impressive scores on traditional benchmarks, an increasing number of researchers are becoming concerned about benchmark data leakage during pre-training, commonly known as the data contamination problem. To ensure fair evaluation, recent benchmarks release only the training and validation sets, keeping the test set labels closed-source. They require anyone wishing to evaluate his language model to submit the model's predictions for centralized processing and then publish the model's result on their leaderboard. However, this submission process is inefficient and prevents effective error analysis. To address this issue, we propose to variabilize benchmarks and evaluate language models dynamically. Specifically, we extract variables from each test case and define a value range for each variable. For each evaluation, we sample new values from these value ranges to create unique test cases, thus ensuring a fresh evaluation each time. We applied this variable perturbation method to four datasets: GSM8K, ARC, CommonsenseQA, and TruthfulQA, which cover mathematical generation and multiple-choice tasks. Our experimental results demonstrate that this approach provides a more accurate assessment of the true capabilities of language models, effectively mitigating the contamination problem.