Arena-Lite: Efficient and Reliable Large Language Model Evaluation via Tournament-Based Direct Comparisons

TL;DR

Arena-Lite introduces a tournament-based evaluation method for large language models that improves reliability and reduces comparisons by directly pitting models against each other, bypassing traditional baseline comparisons.

Contribution

It proposes a novel tournament structure for LLM evaluation that enhances reliability and efficiency over existing baseline-mediated benchmarks.

Findings

Higher reliability in system rankings with fewer comparisons.

Effective even with smaller datasets or weaker judges.

Demonstrated through controlled and empirical experiments.

Abstract

As Large Language Models (LLMs) expand across domains, LLM judges have become essential for systems evaluation. Current benchmarks typically compare system outputs against baselines. This baseline-mediated approach, though convenient, yields lower reliability than direct comparison between systems. We propose Arena-Lite which integrates tournament structure on top of head-to-head comparison. The application of a tournament structure and direct comparison eliminates the need for baseline outputs, reduces the number of required comparisons, and allows higher reliability in system rankings. We conducted two experiments: (1) controlled stochastic modeling and (2) empirical validation with a real LLM judge. Those experiments collectively demonstrate that Arena-Lite consistently achieves higher reliability with fewer comparisons, even with smaller datasets or weaker judges. We release an easy-to-use web demonstration and code to foster adoption of Arena-Lite, streamlining model selection across research and industry communities. Arena-Lite demo and code are available on \href{https://huggingface.co/spaces/NCSOFT/ArenaLite}{https://huggingface.co/spaces/NCSOFT/ArenaLite}