Using tournaments to calculate AUROC for zero-shot classification with LLMs

TL;DR

This paper introduces a novel method that uses tournament-style pairwise comparisons and Elo ratings to evaluate zero-shot classification performance of large language models, providing a fairer and more informative metric.

Contribution

The work proposes a new approach transforming zero-shot tasks into pairwise comparisons with Elo ratings, improving evaluation fairness and informativeness.

Findings

The Elo-based method yields more reliable confidence ordering of instances.

Scheduling algorithms can reduce the number of comparisons needed.

The proposed approach enhances classification performance assessment.

Abstract

Large language models perform surprisingly well on many zero-shot classification tasks, but are difficult to fairly compare to supervised classifiers due to the lack of a modifiable decision boundary. In this work, we propose and evaluate a method that transforms binary classification tasks into pairwise comparisons between instances within a dataset, using LLMs to produce relative rankings of those instances. Repeated pairwise comparisons can be used to score instances using the Elo rating system (used in chess and other competitions), inducing a confidence ordering over instances in a dataset. We evaluate scheduling algorithms for their ability to minimize comparisons, and show that our proposed algorithm leads to improved classification performance, while also providing more information than traditional zero-shot classification.