Ranking Unraveled: Recipes for LLM Rankings in Head-to-Head AI Combat

TL;DR

This paper examines the effectiveness of pairwise ranking methods for evaluating large language models through head-to-head comparisons, providing insights and guidelines for improving ranking accuracy and robustness.

Contribution

It formally defines principles for effective LLM ranking and evaluates various algorithms' robustness, offering practical guidelines for their application.

Findings

Ranking accuracy depends on specific evaluation contexts.

Certain algorithms demonstrate higher robustness under resource constraints.

Guidelines for selecting appropriate ranking methods are provided.

Abstract

Deciding which large language model (LLM) to use is a complex challenge. Pairwise ranking has emerged as a new method for evaluating human preferences for LLMs. This approach entails humans evaluating pairs of model outputs based on a predefined criterion. By collecting these comparisons, a ranking can be constructed using methods such as Elo. However, applying these algorithms as constructed in the context of LLM evaluation introduces several challenges. In this paper, we explore the effectiveness of ranking systems for head-to-head comparisons of LLMs. We formally define a set of fundamental principles for effective ranking and conduct a series of extensive evaluations on the robustness of several ranking algorithms in the context of LLMs. Our analysis uncovers key insights into the factors that affect ranking accuracy and efficiency, offering guidelines for selecting the most appropriate methods based on specific evaluation contexts and resource constraints.