An Interpretable and Scalable Framework for Evaluating Large Language Models

TL;DR

This paper introduces a scalable, interpretable framework for evaluating large language models using a reformulation of Item Response Theory, providing faster and more insightful assessments than existing methods.

Contribution

It presents a novel, efficient approach based on majorization-minimization for large-scale LLM evaluation, improving stability, interpretability, and computational speed.

Findings

Achieves significant speedups over existing methods.

Maintains or improves estimation accuracy.

Provides insights into item difficulty and discrimination.

Abstract

Evaluation of large language models (LLMs) is increasingly critical, yet standard benchmarking methods rely on average accuracy, overlooking both the inherent stochasticity of LLM outputs and the heterogeneity of benchmark items. Item Response Theory (IRT) offers a principled framework for modeling latent model abilities and item characteristics, but conventional methods are computationally expensive and numerically unstable, limiting large-scale implementations. To address these challenges, we propose an interpretable and scalable framework for LLM evaluation based on the majorization-minimization principle. Our approach reformulates the problem as a sequence of constrained matrix factorization subproblems, enabling stable and efficient parameter estimation with theoretical guarantees for identifiability and convergence. Experiments on synthetic and real-world datasets, including MATH-500 and six Open LLM Leaderboard benchmarks, demonstrate that our method achieves superior scalability and interpretability. It delivers orders-of-magnitude speedups over competing methods while maintaining comparable or even higher estimation accuracy. Our results align with established scaling laws and offer insights into item difficulty and discrimination, informing more principled benchmark design.