Distribution-Calibrated Inference time compute for Thinking LLM-as-a-Judge

TL;DR

This paper introduces a distribution-calibrated aggregation method for large language models acting as judges, improving the reliability of pairwise preference evaluations by modeling preferences with a Bradley-Terry-Davidson approach and leveraging multiple samples.

Contribution

It proposes a novel distribution-calibrated aggregation scheme for inference-time compute, enhancing the accuracy and reliability of LLM-based judgments in preference evaluation.

Findings

Reduces mean absolute error in preference ratings

Increases pairwise accuracy over standard baselines

Matches or exceeds human raters in consensus evaluation

Abstract

Thinking Large Language Models (LLMs) used as judges for pairwise preferences remain noisy at the single-sample level, and common aggregation rules (majority vote, soft self-consistency, or instruction-based self-aggregation) are inconsistent when ties are allowed. We study inference-time compute (ITC) for evaluators that generate n independent thinking-rating samples per item, and propose a principled, distribution-calibrated aggregation scheme. Our method models three-way preferences with a Bradley-Terry-Davidson formulation on rating counts, leveraging both polarity (margin among non-ties) and decisiveness (non-tie rate) to distinguish narrow margins from strong consensus. Across various evaluation benchmarks, our approach consistently reduces MAE and increases pairwise accuracy versus standard baselines, and when evaluated against human-consensus meta-labels, matches or exceeds individual human raters. These results show that carefully allocating ITC and aggregating with distribution-aware methods turns noisy individual model judgments into reliable ratings for evaluation.