Knockout LLM Assessment: Using Large Language Models for Evaluations through Iterative Pairwise Comparisons

TL;DR

This paper introduces Knockout Assessment, a novel iterative pairwise comparison method using LLMs as judges, which enhances the accuracy of evaluations like exam scoring and machine translation by mimicking tournament-style rankings.

Contribution

It proposes a knockout tournament framework for LLM-based evaluation, enabling the development of a global ranking perspective and improving correlation with human assessments.

Findings

Increases Pearson correlation with expert scores by 0.07 on average.

Enhances scoring accuracy in university exams and machine translation evaluations.

Aligns LLM assessments more closely with human judgments.

Abstract

Large Language Models (LLMs) have shown to be effective evaluators across various domains such as machine translations or the scientific domain. Current LLM-as-a-Judge approaches rely mostly on individual assessments or a single round of pairwise assessments, preventing the judge LLM from developing a global ranking perspective. To address this, we present Knockout Assessment, an LLM-asa Judge method using a knockout tournament system with iterative pairwise comparisons. Experiments across three LLMs on two datasets show that knockout assessment improves scoring accuracy, increasing Pearson correlation with expert evaluations by 0.07 on average for university-level exam scoring and machine translation evaluations, aligning LLM assessments more closely with human scoring.