Prompt-to-Leaderboard

TL;DR

Prompt-to-Leaderboard (P2L) introduces prompt-specific leaderboards for language models, enabling nuanced evaluation, personalized query routing, and better understanding of model strengths, outperforming traditional aggregated metrics.

Contribution

We propose a novel method that creates prompt-dependent leaderboards by training LLMs to predict human preferences, capturing performance variations overlooked by traditional metrics.

Findings

P2L better captures nuanced model performance than averaged leaderboards.

P2L's prompt-specific evaluations follow a power law scaling.

A P2L-based router achieved #1 on Chatbot Arena in January 2025.

Abstract

Large language model (LLM) evaluations typically rely on aggregated metrics like accuracy or human preference, averaging across users and prompts. This averaging obscures user- and prompt-specific variations in model performance. To address this, we propose Prompt-to-Leaderboard (P2L), a method that produces leaderboards specific to a prompt. The core idea is to train an LLM taking natural language prompts as input to output a vector of Bradley-Terry coefficients which are then used to predict the human preference vote. The resulting prompt-dependent leaderboards allow for unsupervised task-specific evaluation, optimal routing of queries to models, personalization, and automated evaluation of model strengths and weaknesses. Data from Chatbot Arena suggest that P2L better captures the nuanced landscape of language model performance than the averaged leaderboard. Furthermore, our findings suggest that P2L's ability to produce prompt-specific evaluations follows a power law scaling similar to that observed in LLMs themselves. In January 2025, the router we trained based on this methodology achieved the #1 spot on the Chatbot Arena leaderboard. Our code is available on GitHub at https://github.com/lmarena/p2l.