Strategic Candidacy in Generative AI Arenas

TL;DR

This paper examines how model producers can exploit noisy AI arena rankings by submitting clones, and introduces YRWR, a new ranking mechanism that is approximately clone-robust and improves accuracy.

Contribution

The paper introduces YRWR, a novel ranking mechanism that mitigates clone exploitation and enhances ranking accuracy in generative AI arenas.

Findings

Producers can benefit from submitting clones under current ranking methods.

YRWR is approximately clone-robust, limiting benefits from cloning.

Simulations show YRWR improves ranking accuracy even with producer misranking.

Abstract

AI arenas, which rank generative models from pairwise preferences of users, are a popular method for measuring the relative performance of models in the course of their organic use. Because rankings are computed from noisy preferences, there is a concern that model producers can exploit this randomness by submitting many models (e.g., multiple variants of essentially the same model) and thereby artificially improve the rank of their top models. This can lead to degradations in the quality, and therefore the usefulness, of the ranking. In this paper, we begin by establishing, both theoretically and in simulations calibrated to data from the platform Arena (formerly LMArena, Chatbot Arena), conditions under which producers can benefit from submitting clones when their goal is to be ranked highly. We then propose a new mechanism for ranking models from pairwise comparisons, called You-Rank-We-Rank (YRWR). It requires that producers submit rankings over their own models and uses these rankings to correct statistical estimates of model quality. We prove that this mechanism is approximately clone-robust, in the sense that a producer cannot improve their rank much by doing anything other than submitting each of their unique models exactly once. Moreover, to the extent that model producers are able to correctly rank their own models, YRWR improves overall ranking accuracy. In further simulations, we show that indeed the mechanism is approximately clone-robust and quantify improvements to ranking accuracy, even under producer misranking.