Training on the Test Task Confounds Evaluation and Emergence

TL;DR

This paper highlights how training on evaluation tasks can bias model assessments and the perceived emergence of capabilities, proposing a method to mitigate this effect for fairer benchmarking.

Contribution

It introduces a method to adjust for training on test tasks in evaluations and demonstrates its impact on model performance and emergent behavior claims.

Findings

Training on test tasks confounds evaluation results.

Adjusting for test task training alters perceived model capabilities.

Emergent behaviors diminish when models are uniformly trained on test data.

Abstract

We study a fundamental problem in the evaluation of large language models that we call training on the test task. Unlike wrongful practices like training on the test data, leakage, or data contamination, training on the test task is not a malpractice. Rather, the term describes a growing set of practices that utilize knowledge about evaluation tasks at training time. We demonstrate that training on the test task confounds both relative model evaluations and claims about emergent capabilities. We argue that the seeming superiority of one model family over another may be explained by a different degree of training on the test task. To this end, we propose an effective method to adjust for the effect of training on the test task on benchmark evaluations. Put simply, to fine-tune each model under comparison on the same task-relevant data prior to evaluation. We then show that instances of emergent behavior disappear gradually as models train on the test task. Our work promotes a new perspective on the evaluation of large language models, with broad implications for benchmarking and the study of emergent capabilities.