Resurrecting saturated LLM benchmarks with adversarial encoding

TL;DR

This paper demonstrates that modifying benchmark questions with pairing and additional answer options can reveal the true capabilities of large language models by preventing performance saturation.

Contribution

It introduces a method to resurface the difficulty of existing benchmarks by adversarially modifying questions, thus providing a more accurate assessment of LLMs.

Findings

Modified benchmarks show reduced model performance, indicating previous saturation.

The approach can effectively resurrect and extend the utility of older benchmarks.

Capable models' performance is more accurately reflected after modifications.

Abstract

Recent work showed that small changes in benchmark questions can reduce LLMs' reasoning and recall. We explore two such changes: pairing questions and adding more answer options, on three benchmarks: WMDP-bio, GPQA, and MMLU variants. We find that for more capable models, these predictably reduce performance, essentially heightening the performance ceiling of a benchmark and unsaturating it again. We suggest this approach can resurrect old benchmarks.