Becoming Experienced Judges: Selective Test-Time Learning for Evaluators

TL;DR

This paper introduces a novel framework called Learning While Evaluating (LWE) that enables large language model evaluators to improve their assessment quality during inference by self-refinement, especially focusing on challenging cases, without additional training.

Contribution

The paper proposes LWE and its selective variant, allowing evaluators to adaptively improve during inference through self-generated feedback and selective updates, enhancing evaluation accuracy.

Findings

Selective LWE outperforms strong baselines on benchmarks.

Evaluators improve during sequential testing.

Focusing on difficult cases yields better learning efficiency.

Abstract

Automatic evaluation with large language models, commonly known as LLM-as-a-judge, is now standard across reasoning and alignment tasks. Despite evaluating many samples in deployment, these evaluators typically (i) treat each case independently, missing the opportunity to accumulate experience, and (ii) rely on a single fixed prompt for all cases, neglecting the need for sample-specific evaluation criteria. We introduce Learning While Evaluating (LWE), a framework that allows evaluators to improve sequentially at inference time without requiring training or validation sets. LWE maintains an evolving meta-prompt that (i) produces sample-specific evaluation instructions and (ii) refines itself through self-generated feedback. Furthermore, we propose Selective LWE, which updates the meta-prompt only on self-inconsistent cases, focusing computation where it matters most. This selective approach retains the benefits of sequential learning while being far more cost-effective. Across two pairwise comparison benchmarks, Selective LWE outperforms strong baselines, empirically demonstrating that evaluators can improve during sequential testing with a simple selective update, learning most from the cases they struggle with.