Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theory

TL;DR

This paper critically examines the effectiveness of large language model benchmarks, introduces an improved IRT-based framework called PSN-IRT, and demonstrates its ability to better evaluate and optimize benchmark design for LLMs.

Contribution

It introduces PSN-IRT, an enhanced IRT framework for more accurate LLM evaluation, and provides a comprehensive analysis revealing shortcomings of current benchmarks.

Findings

Current benchmarks show significant measurement shortcomings.

PSN-IRT can construct smaller, more aligned benchmarks.

Enhanced evaluation leads to better understanding of LLM capabilities.

Abstract

The evaluation of large language models (LLMs) via benchmarks is widespread, yet inconsistencies between different leaderboards and poor separability among top models raise concerns about their ability to accurately reflect authentic model capabilities. This paper provides a critical analysis of benchmark effectiveness, examining mainstream prominent LLM benchmarks using results from diverse models. We first propose Pseudo-Siamese Network for Item Response Theory (PSN-IRT), an enhanced Item Response Theory framework that incorporates a rich set of item parameters within an IRT-grounded architecture. PSN-IRT can be utilized for accurate and reliable estimations of item characteristics and model abilities. Based on PSN-IRT, we conduct extensive analysis on 11 LLM benchmarks comprising 41,871 items, revealing significant and varied shortcomings in their measurement quality. Furthermore, we demonstrate that leveraging PSN-IRT is able to construct smaller benchmarks while maintaining stronger alignment with human preference.