Aligning Language Model Benchmarks with Pairwise Preferences

TL;DR

This paper introduces BenchAlign, a method to automatically update language model benchmarks to better predict pairwise model preferences, aligning them with human-like utility and preferences.

Contribution

We propose BenchAlign, the first approach to learn preference-aligned benchmark question weightings using model performance and pairwise preferences, improving ranking accuracy.

Findings

Aligned benchmarks accurately rank unseen models by human preferences

BenchAlign remains interpretable across different model sizes

Our method bridges the gap between benchmarks and real-world utility

Abstract

Language model benchmarks are pervasive and computationally-efficient proxies for real-world performance. However, many recent works find that benchmarks often fail to predict real utility. Towards bridging this gap, we introduce benchmark alignment, where we use limited amounts of information about model performance to automatically update offline benchmarks, aiming to produce new static benchmarks that predict model pairwise preferences in given test settings. We then propose BenchAlign, the first solution to this problem, which learns preference-aligned weight- ings for benchmark questions using the question-level performance of language models alongside ranked pairs of models that could be collected during deployment, producing new benchmarks that rank previously unseen models according to these preferences. Our experiments show that our aligned benchmarks can accurately rank unseen models according to models of human preferences, even across different sizes, while remaining interpretable. Overall, our work provides insights into the limits of aligning benchmarks with practical human preferences, which stands to accelerate model development towards real utility.