Improving LLM Leaderboards with Psychometrical Methodology

TL;DR

This paper proposes applying psychometric methodologies, traditionally used in human assessments, to improve the evaluation and ranking of large language models on leaderboards, leading to more robust and meaningful comparisons.

Contribution

It introduces psychometric techniques into LLM benchmarking, replacing simplistic aggregation methods with more rigorous evaluation approaches.

Findings

Psychometric methods improve LLM ranking robustness.

Psychometric evaluation provides more meaningful performance insights.

Comparison shows advantages over naive averaging methods.

Abstract

The rapid development of large language models (LLMs) has necessitated the creation of benchmarks to evaluate their performance. These benchmarks resemble human tests and surveys, as they consist of sets of questions designed to measure emergent properties in the cognitive behavior of these systems. However, unlike the well-defined traits and abilities studied in social sciences, the properties measured by these benchmarks are often vaguer and less rigorously defined. The most prominent benchmarks are often grouped into leaderboards for convenience, aggregating performance metrics and enabling comparisons between models. Unfortunately, these leaderboards typically rely on simplistic aggregation methods, such as taking the average score across benchmarks. In this paper, we demonstrate the advantages of applying contemporary psychometric methodologies - originally developed for human tests and surveys - to improve the ranking of large language models on leaderboards. Using data from the Hugging Face Leaderboard as an example, we compare the results of the conventional naive ranking approach with a psychometrically informed ranking. The findings highlight the benefits of adopting psychometric techniques for more robust and meaningful evaluation of LLM performance.