Elo Uncovered: Robustness and Best Practices in Language Model Evaluation

TL;DR

This paper critically examines the use of the Elo rating system for evaluating large language models, highlighting its limitations in reliability and transitivity, and proposes guidelines to improve evaluation robustness.

Contribution

It provides an in-depth analysis of Elo's behavior in LLM evaluation, revealing volatility issues and offering practical recommendations for more reliable assessment methods.

Findings

Elo scores can be volatile and inconsistent.

Varying Elo hyperparameters affects evaluation reliability.

Current Elo-based evaluations may not satisfy key axioms.

Abstract

In Natural Language Processing (NLP), the Elo rating system, originally designed for ranking players in dynamic games such as chess, is increasingly being used to evaluate Large Language Models (LLMs) through "A vs B" paired comparisons. However, while popular, the system's suitability for assessing entities with constant skill levels, such as LLMs, remains relatively unexplored. We study two fundamental axioms that evaluation methods should adhere to: reliability and transitivity. We conduct extensive evaluation of Elo behaviour, illustrating that individual Elo computations exhibit volatility and delving into the impact of varying the Elo rating system's hyperparameters. We show that these axioms are not always satisfied raising questions about the reliability of current comparative evaluations of LLMs. If the current use of Elo scores is intended to substitute the costly head-to-head comparison of LLMs, it is crucial to ensure the ranking is as robust as possible. Guided by the axioms, our findings offer concrete guidelines for enhancing the reliability of LLM evaluation methods, suggesting a need for reassessment of existing comparative approaches.