Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judge

TL;DR

This paper introduces a crowd-based evaluation method for LLM-as-a-Judge that improves the depth and reliability of judgments by incorporating additional crowd responses, leading to more accurate and comprehensive assessments.

Contribution

It proposes a novel crowd-based comparative evaluation approach that enhances the depth, accuracy, and reliability of LLM judgments beyond existing majority voting methods.

Findings

Achieves an average accuracy gain of 6.7% across five benchmarks.

Produces higher-quality, more comprehensive chain-of-thoughts.

Improves evaluation accuracy as inference scales.

Abstract

LLM-as-a-Judge, which generates chain-of-thought (CoT) judgments, has become a widely adopted auto-evaluation method. However, its reliability is compromised by the CoT reasoning's inability to capture comprehensive and deeper details, often leading to incomplete outcomes. Existing methods mainly rely on majority voting or criteria expansion, which is insufficient to address the limitation in CoT. We propose Crowd-based Comparative Evaluation, which introduces additional crowd responses to compare with the candidate responses, thereby exposing deeper and more comprehensive details within the candidate responses. This process effectively guides LLM-as-a-Judge to provide a more detailed CoT judgment. Extensive experiments demonstrate that our approach enhances evaluation reliability, achieving an average accuracy gain of 6.7% across five benchmarks. Moreover, our method produces higher-quality CoTs that facilitate judge distillation and exhibit superior performance in rejection sampling for supervised fine-tuning (SFT), referred to as crowd rejection sampling, thereby enabling more efficient SFT. Our analysis confirms that CoTs generated by ours are more comprehensive and of higher quality, and evaluation accuracy improves as inference scales.