Position: AI Competitions Provide the Gold Standard for Empirical Rigor in GenAI Evaluation

TL;DR

This paper argues that AI competitions are the most rigorous and effective method for evaluating Generative AI models due to their proven strategies against issues like leakage and contamination, which are prevalent in traditional evaluation methods.

Contribution

It highlights the potential of AI competitions to serve as the gold standard for empirical evaluation of GenAI, addressing current evaluation challenges.

Findings

AI competitions effectively combat leakage and contamination.

Traditional evaluation methods are insufficient for modern GenAI models.

AI competitions can improve the rigor of GenAI evaluation.

Abstract

In this position paper, we observe that empirical evaluation in Generative AI is at a crisis point since traditional ML evaluation and benchmarking strategies are insufficient to meet the needs of evaluating modern GenAI models and systems. There are many reasons for this, including the fact that these models typically have nearly unbounded input and output spaces, typically do not have a well defined ground truth target, and typically exhibit strong feedback loops and prediction dependence based on context of previous model outputs. On top of these critical issues, we argue that the problems of leakage and contamination are in fact the most important and difficult issues to address for GenAI evaluations. Interestingly, the field of AI Competitions has developed effective measures and practices to combat leakage for the purpose of counteracting cheating by bad actors within a competition setting. This makes AI Competitions an especially valuable (but underutilized) resource. Now is time for the field to view AI Competitions as the gold standard for empirical rigor in GenAI evaluation, and to harness and harvest their results with according value.