Who Defines "Best"? Towards Interactive, User-Defined Evaluation of LLM Leaderboards

TL;DR

This paper introduces an interactive visualization tool for LLM leaderboards, enabling users to customize evaluation criteria and better understand model performance across diverse prompts.

Contribution

It presents a novel interactive interface allowing users to define and explore personalized evaluation priorities for LLMs, addressing limitations of traditional leaderboards.

Findings

Dataset is heavily skewed toward certain topics.

Model rankings vary across different prompt slices.

Preference-based judgments can blur evaluation scope.

Abstract

LLM leaderboards are widely used to compare models and guide deployment decisions. However, leaderboard rankings are shaped by evaluation priorities set by benchmark designers, rather than by the diverse goals and constraints of actual users and organizations. A single aggregate score often obscures how models behave across different prompt types and compositions. In this work, we conduct an in-depth analysis of the dataset used in the LMArena (formerly Chatbot Arena) benchmark and investigate this evaluation challenge by designing an interactive visualization interface as a design probe. Our analysis reveals that the dataset is heavily skewed toward certain topics, that model rankings vary across prompt slices, and that preference-based judgments are used in ways that blur their intended scope. Building on this analysis, we introduce a visualization interface that allows users to define their own evaluation priorities by selecting and weighting prompt slices and to explore how rankings change accordingly. A qualitative study suggests that this interactive approach improves transparency and supports more context-specific model evaluation, pointing toward alternative ways to design and use LLM leaderboards.