RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty

TL;DR

RankLLM introduces a novel framework that quantifies question difficulty and model competency, enabling more nuanced evaluation of large language models by considering question challenge levels.

Contribution

It presents a new bidirectional score propagation method to assess LLMs based on question difficulty, improving evaluation granularity over existing benchmarks.

Findings

Achieves 90% agreement with human judgments

Outperforms strong baselines like IRT

Demonstrates stability, fast convergence, and efficiency

Abstract

Benchmarks establish a standardized evaluation framework to systematically assess the performance of large language models (LLMs), facilitating objective comparisons and driving advancements in the field. However, existing benchmarks fail to differentiate question difficulty, limiting their ability to effectively distinguish models' capabilities. To address this limitation, we propose RankLLM, a novel framework designed to quantify both question difficulty and model competency. RankLLM introduces difficulty as the primary criterion for differentiation, enabling a more fine-grained evaluation of LLM capabilities. RankLLM's core mechanism facilitates bidirectional score propagation between models and questions. The core intuition of RankLLM is that a model earns a competency score when it correctly answers a question, while a question's difficulty score increases when it challenges a model. Using this framework, we evaluate 30 models on 35,550 questions across multiple domains. RankLLM achieves 90% agreement with human judgments and consistently outperforms strong baselines such as IRT. It also exhibits strong stability, fast convergence, and high computational efficiency, making it a practical solution for large-scale, difficulty-aware LLM evaluation.