am-ELO: A Stable Framework for Arena-based LLM Evaluation

TL;DR

This paper introduces am-ELO, a stable arena-based evaluation framework for large language models that improves upon existing ELO-based methods by incorporating annotator abilities and using MLE for stability.

Contribution

It proposes a novel am-ELO framework that enhances model evaluation stability by modeling annotator abilities and replacing iterative updates with MLE, with theoretical and empirical validation.

Findings

am-ELO provides more stable and reliable LLM evaluations.

Theoretical proof confirms the consistency of the MLE approach.

Experiments show improved robustness over existing methods.

Abstract

Arena-based evaluation is a fundamental yet significant evaluation paradigm for modern AI models, especially large language models (LLMs). Existing framework based on ELO rating system suffers from the inevitable instability problem due to ranking inconsistency and the lack of attention to the varying abilities of annotators. In this paper, we introduce a novel stable arena framework to address these issues by enhancing the ELO Rating System. Specifically, we replace the iterative update method with a Maximum Likelihood Estimation (MLE) approach, m-ELO, and provide theoretical proof of the consistency and stability of the MLE approach for model ranking. Additionally, we proposed the am-ELO, which modify the Elo Rating's probability function to incorporate annotator abilities, enabling the simultaneous estimation of model scores and annotator reliability. Experiments demonstrate that this method ensures stability, proving that this framework offers a more robust, accurate, and stable evaluation method for LLMs.