League of LLMs: A Benchmark-Free Paradigm for Mutual Evaluation of Large Language Models

TL;DR

The paper introduces LOL, a benchmark-free, multi-round mutual evaluation framework for LLMs that addresses evaluation challenges like data contamination and subjectivity, providing more reliable and insightful assessments.

Contribution

It proposes a novel league-based evaluation paradigm integrating four core criteria, enabling effective LLM ranking and revealing new empirical insights.

Findings

LOL effectively distinguishes LLM capabilities with high ranking stability.

Memorization-based answering behaviors are observed in some models.

Higher in-family scores are found in the OpenAI model family.

Abstract

Although large language models (LLMs) have shown exceptional capabilities across a wide range of tasks, reliable evaluation remains a critical challenge due to data contamination, opaque operation, and subjective preferences. To address these issues, we propose League of LLMs (LOL), a novel benchmark-free evaluation paradigm that organizes multiple LLMs into a self-governed league for multi-round mutual evaluation. LOL integrates four core criteria (dynamic, transparent, objective, and professional) to mitigate key limitations of existing paradigms. Experiments on eight mainstream LLMs in mathematics and programming demonstrate that LOL can effectively distinguish LLM capabilities while maintaining high internal ranking stability (Top-$k$ consistency $= 70.7\%$). Beyond ranking, LOL reveals empirical findings that are difficult for traditional paradigms to capture. For instance, ``memorization-based answering'' behaviors are observed in some models, and higher in-family scores are found in the OpenAI model family ($\Delta = 9$, $p < 0.05$). Finally, we make our framework and code publicly available as a valuable complement to the current LLM evaluation ecosystem.