Pretraining on the Test Set Is No Longer All You Need: A Debate-Driven Approach to QA Benchmarks

TL;DR

This paper introduces a debate-driven evaluation framework transforming QA benchmarks into adversarial debates, improving robustness and reducing data contamination issues in assessing language models.

Contribution

It presents a systematic debate-based assessment pipeline and a public benchmark demonstrating enhanced evaluation robustness and scalability for language models.

Findings

Debate-based evaluation increases difficulty and penalizes shallow memorization.

Models fine-tuned on test questions perform worse in debates, indicating robustness.

Even weaker judges can reliably evaluate stronger debaters.

Abstract

As frontier language models increasingly saturate standard QA benchmarks, concerns about data contamination, memorization, and escalating dataset creation costs persist. We propose a debate-driven evaluation paradigm that transforms any existing QA dataset into structured adversarial debates--where one model is given the official answer to defend, and another constructs and defends an alternative answer--adjudicated by a judge model blind to the correct solution. By forcing multi-round argumentation, this approach substantially increases difficulty while penalizing shallow memorization, yet reuses QA items to reduce curation overhead. We make two main contributions: (1) an evaluation pipeline to systematically convert QA tasks into debate-based assessments, and (2) a public benchmark that demonstrates our paradigm's effectiveness on a subset of MMLU-Pro questions, complete with standardized protocols and reference models. Empirical results validate the robustness of the method and its effectiveness against data contamination--a Llama 3.1 model fine-tuned on test questions showed dramatic accuracy improvements (50% -> 82%) but performed worse in debates. Results also show that even weaker judges can reliably differentiate stronger debaters, highlighting how debate-based evaluation can scale to future, more capable systems while maintaining a fraction of the cost of creating new benchmarks. Overall, our framework underscores that "pretraining on the test set is no longer all you need," offering a sustainable path for measuring the genuine reasoning ability of advanced language models.